Evaluation of research in biomedical ontologies

Briefings in Bioinformatics

Volumn 14, Issue 6, 2013, Pages 696-712

  Hoehndorf, Robert ^a   Dumontier, Michel ^b   Gkoutos, Georgios V ^c  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^b CARLETON UNIVERSITY   (Canada)

^c ABERYSTWYTH UNIVERSITY   (United Kingdom)

Author keywords

Biomedical ontology; Evaluation criteria; Ontology evaluation; Ontology based applications; Quantitative biology

Indexed keywords

ARTICLE; BIOLOGICAL ONTOLOGY; EVALUATION CRITERIA; EVALUATION STUDY; MEDICAL ONTOLOGY; MEDICAL RESEARCH; ONTOLOGY EVALUATION; ONTOLOGY-BASED APPLICATIONS; QUANTITATIVE BIOLOGY;

BIOMEDICAL ONTOLOGY; EVALUATION CRITERIA; ONTOLOGY EVALUATION; ONTOLOGY-BASED APPLICATIONS; QUANTITATIVE BIOLOGY;

BIOLOGICAL ONTOLOGIES; BIOMEDICAL RESEARCH;

EID: 84887775514     PISSN: 14675463     EISSN: 14774054     Source Type: Journal    
DOI: 10.1093/bib/bbs053     Document Type: Article

References (124)

1. Smith B, Ashburner M, Rosse C, et al. The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotech 2007;25:1251-5.
2. Merrill GH. Ontological realism: methodology or misdirection? Appl Ontol 2010;5:79-108.
3. Smith B, Ceusters W. Ontological realism: a methodology for coordinated evolution of scientific ontologies. Appl Ontol 2010;5:139-88.
4. Obrst L, Ceusters W, Mani I, et al. The Evaluation of Ontologies: Toward Improved Semantic Interoperability. In: Baker CJO, Cheung K-H, (eds). Semantic Web: Revolutionizing Knowledge Discovery in the Life Sciences 2007. Springer Science+Business Media, New York, NY, U.S.A.
5. Smith B. Proceeding of the 2008Conference on Formal Ontology in Information Systems: Proceedings of the Fifth International Conference (FOIS 2008). pp. 21-35. IOS Press, Amsterdam, The Netherlands, 2008.
6. Rector AL, Qamar R, Marley T. Binding ontologies and coding systems to electronic health records and messages. Appl Ontol 2009;4:51-69.
7. Rogers J, Rector AL. Terminological Systems: bridging the generation gap. Proceedings of the AMIA Annual Fall Symposium 1997;1997:610-4.
8. Fung K, Bodenreider O. Knowledge representation and ontologies. In: Richesson R, Andrews J, (eds). Clinical Research Informatics. London, UK: Springer Verlag, 2012; 255-75.
9. Nohama P, Pacheco E, Andrade R, et al. Quality issues in thesaurus building: a case study from the medical domain. BrazJ Biomed Eng 2012;28:11-22.
10. Freimer N, Sabatti C. The human phenome project. Nat Genet 2003;34:15-21.
11. Ashburner M, Ball CA, Blake JA, et al. Gene ontology: tool for the unification of biology. Nat Genet 2000; 25:25-9.
12. Robinson GE, Hackett KJ, Purcell-Miramontes M, et al. Creating a buzz about insect genomes. Science 2011;331: 1386.
13. Genome 10K Community of Scientists. Genome 10K: a proposal to obtain whole-genome sequence for 10000 vertebrate species. JHered 2009;100:659-74.
14. Collins FS, Finnell RH, Rossant J, et al. A new partner for the international knockout mouse consortium. Cell 2007; 129:235.
15. Skarnes WC, Rosen B, West AP, et al. A conditional knockout resource for the genome-wide study of mouse gene function. Nature 2011;474:337-42.
16. Parkinson H, Sarkans U, Kolesnikov N, et al. ArrayExpress update: an archive of microarray and high-throughput sequencing-based functional genomics experiments. Nucleic Acids Res 2011;39:D1002-4.
17. Munos B. Lessons from 60 years of pharmaceutical innovation. Nat RevDrugDiscov 2009;8:959-68.
18. Ekins S, Williams AJ, Krasowski MD, et al. In silico repositioning of approved drugs for rare and neglected diseases. Drug DiscovToday 2011;16:298-310.
19. Horrocks I. OBO Flat File Format Syntax and Semantics and Mapping to OWL Web Ontology Language Tech. Rep. Manchester, UK: University of Manchester, 2007. http:// www.cs.man.ac.uk/~horrocks/obo/.
20. Golbreich C, Horrocks I. The OBO to OWL mapping, GO to OWL 1.1!. In: Golbreich C, Kalyanpur A, Parsia B, (eds). Proceedings of OWL: Experiences and Directions 2007 (OWLED-2007). Aachen, Germany: CEUR-WS.org, 2007.
21. Smith B, Ceusters W, Klagges B, et al. Relations in biomedical ontologies. Genome Biol 2005;6:R46.
22. Hoehndorf R, Oellrich A, Dumontier M, et al. Relations as patterns: bridging the gap between OBO and OWL. BMC Bioinformatics 2010;11:441+.
23. Subramanian A, Tamayo P, Mootha VK, et al. Gene set enrichment analysis: a knowledge-based approach for interpreting genome-wide expression profiles. ProcNatlAcad SciUSA 2005;102:15545-50.
24. Prufer K, Muetzel B, Do H-H, et al. FUNC: a package for detecting significant associations between gene sets and ontological annotations. BMC Bioinformatics 2007;8:41+.
25. Mootha VK, Lindgren CM, Eriksson K-F, et al. PGC-1alpharesponsive genes involved in oxidative phosphorylation are coordinately downregulated in human diabetes. Nat Genet 2003;34:267-73.
26. LePendu P, Musen M, Shah N. Enabling enrichment analysis with the human disease ontology. JBiomed Inform 2011; 44(Suppl 1):S31-8.
27. Resnik P. Semantic similarity in a taxonomy: an information-based measure and its application to problems of ambiguity in natural language. JArtif Intell Res 1999;11: 95-130.
28. Du Plessis L, Kunca N, Dessimoz C. The what, where, how and why of gene ontology: a primer for bioinformaticians. Brief Bioinform 2011;12:723-35.
29. Benabderrahmane S, Smail-Tabbone M, Poch O, et al. IntelliGO: a new vector-based semantic similarity measure including annotation origin. BMC Bioinformatics 2010;11: 588.
30. Schlicker A, Lengauer T, Albrecht M. Improving disease gene prioritization using the semantic similarity of Gene Ontology terms. Bioinformatics 2010;26:i561-7.
31. Guzzi PH, Mina M, Guerra C, et al. Semantic similarity analysis of protein data: assessment with biological features and issues. Brief Bioinform 2011, in press.
32. Bodenreider O, Stevens R. Bio-ontologies: current trends and future directions. Brief Bioinform 2006;7: 256-74.
33. Lord PW, Stevens RD, Brass A, et al. Investigating semantic similarity measures across the Gene Ontology: the relationship between sequence and annotation. Bioinformatics 2003; 19:1275-83.
34. Rubin DL, Shah NH, Noy NF. Biomedical ontologies: a functional perspective. Brief Bioinform 2008;9:75-90.
35. Andronis C, Sharma A, Virvilis V, et al. Literature mining, ontologies and information visualization for drug repurposing. Brief Bioinform 2011;12:357-68.
36. Blaschke C, Leon EA, Krallinger M, et al. Evaluation of BioCreAtIvE assessment of task 2. BMC Bioinformatics 2005;6(Suppl. 1):S16.
37. Leser U, Hakenberg J. What makes a gene name? Named entity recognition in the biomedical literature. Brief Bioinform 2005;6:357-69.
38. Doms A, Schroeder M. GoPubMed: exploring PubMed with the Gene Ontology. Nucleic Acids Res 2005;33:783-6.
39. Yao L, Divoli A, Mayzus I, et al. Benchmarking ontologies: bigger or better? PLoSComput Biol 2011;7:e1001055.
40. Uniprot C. The Universal Protein Resource (UniProt). Nucleic Acids Res 2007;35:D193-7.
41. Bodenreider O, Hayamizu TF, Ringwald M, et al. Of mice and men: aligning mouse and human anatomies. AMIA Annu Symp Proc 2005;2005:61-65.
42. Dameron O, Rubin D, Musen M. Challenges in converting frame-based ontology into OWL: the foundational model of anatomy case-study. AMIA Annu Symp Proc 2005;2005:181.
43. Rosse C, Mejino JLV. A reference ontology for biomedical informatics: the foundational model of anatomy. J Biomed Inform 2003;36:478-500.
44. Haendel M, Neuhaus F, Osumi-Sutherland D, et al. CARO-the common anatomy reference ontology. In: Anatomy Ontologies for Bioinformatics: Principles and Practice. London, UK: Springer Verlag, 2007.
45. Lee RYN, Sternberg PW. Building a cell and anatomy ontology of Caenorhabditis elegans. Comp Funct Genomics 2003;4:121-6.
46. Mungall C, Torniai C, Gkoutos G, et al. Uberon, an integrative multispecies anatomy ontology. Genome Biol 2012; 13:R5.
47. Segerdell E, Bowes J, Pollet N, et al. An ontology for Xenopus anatomy and development. BMC Dev Biol 2008; 8:92.
48. Keseler IM, Collado-Vides J, Santos-Zavaleta A, et al. EcoCyc: a comprehensive database of Escherichia coli biology. Nucleic Acids Res 2011;39:D583-90.
49. Grau B, Horrocks I, Motik B, et al. OWL 2: the next step for OWL. Web Semant Sci ServAgentWorldWideWeb 2008;6: 309-22.
50. Hoehndorf R, Kelso J, Herre H. The ontology of biological sequences. BMC Bioinformatics 2009;10:377+.
51. Hoehndorf R, Batchelor C, Bittner T, et al. The RNA Ontology (RNAO): An ontology for integrating RNA sequence and structure data. Appl Ontol 2011;6: 53-89.
52. Hastings J, Ceusters W, Smith B, et al. InProceedings of the 7th International and Interdisciplinary Conference on Modeling and Using Context. Karlsruhe, Germany: Springer-Verlag, 2011; 119-123.
53. Hilbert D. Axiomatisches Denken. Mathematische Annalen 1918;78:405-15.
54. Barwise J, Etchemendy J. Language, Proof and Logic. Center for the Study of Language and Inf, 2002.
55. Hastings J, Batchelor CR, Steinbeck C, et al. What are chemical structures and their relations? In: Formal Ontology in Information Systems: Proceedings of the Sixth International Conference (FOIS 2010). Amsterdam, The Netherlands: IOS Press, 2010;257-70.
56. Demir E, Cary MP, Paley S, et al. The BioPAX community standard for pathway data sharing. Nat Biotechnol 2010;28: 935-42.
57. Mungall CJ, Bada M, Berardini TZ, et al. Cross-product extensions of the gene ontology. J Biomed Inform 2011;44: 80-6.
58. Hoehndorf R, Dumontier M, Gennari JH, et al. Integrating systems biology models and biomedical ontologies. BMC Syst Biol 2011;5:124+.
59. Ruttenberg A, Clark T, Bug W, et al. Advancing translational research with the semantic web. BMC Bioinformatics 2007;8:S2+.
60. Hoehndorf R, Dumontier M, Oellrich A, et al. Interoperability between biomedical ontologies through relation expansion, upper-level ontologies and automatic reasoning. PLOS One 2011;6:e22006.
61. Mungall CJ. OBO Flat File Format 1.4 Syntax and Semantics [DRAFT] tech. rep. Lawrence Berkeley National Laboratory, 2011. http://berkeleybop.org/~cjm/obo2 owl/obo-syntax.html.
62. Goble C, Stevens R. State of the nation in data integration for bioinformatics. J Biomed Inform 2008;41:687-93.
63. Bada M, Stevens R, Goble C, et al. A short study on the success of the Gene Ontology. Web Semant Sci ServAgents WorldWideWeb 2004;1:235-40.
64. Rector AL. Modularisation of domain ontologies implemented in description logics and related formalisms including OWL. In K-CAP '03: Proceedings of the 2nd International Conference on Knowledge Capture. New York, NY, USA: ACM Press, 2003;121-128.
65. Rector A. Barriers, Approaches and Research Priorities for Integrating Biomedical OntologiesTech. Rep. Manchester, UK: University of Manchester, 2008.
66. Smith B, Williams J, Schulze-Kremer S. The ontology of the Gene Ontology. AMIA Annu Symp Proc 2003;2003: 609-613.
67. Schulz S, Stenzhorn H, Boeker M, et al. Strengths and limitations of formal ontologies in the biomedical domain. RECIIS Electron J Commun Inf Innov Health 2009; 3:31-45.
68. Hoehndorf R, Schofield PN, Gkoutos GV. PhenomeNET: a whole-phenome approach to disease gene discovery. Nucleic Acids Res 2011;39:e119.
69. Mungall CJ, Batchelor C, Eilbeck K. Evolution of the sequence ontology terms and relationships. J Biomed Inform 2011;44:87-93.
70. Schindelman G, Fernandes J, Bastiani C, et al. Worm phenotype ontology: integrating phenotype data within and beyond the C. elegans community. BMC Bioinformatics 2011;12:32.
71. Hoehndorf R, Ngomo A-CN, Kelso J. Applying the functional abnormality ontology pattern to anatomical functions. J Biomed Semant 2010;1:4.
72. Hoehndorf R, Loebe F, Kelso J, et al. Representing default knowledge in biomedical ontologies: application to the integration of anatomy and phenotype ontologies. BMC Bioinformatics 2007;8:377.
73. Rohl J, Jansen L. Representing dispositions. JBiomedSemant 2011;2:S4.
74. Schulz S, Suntisrivaraporn B, Baader F, et al. SNOMED reaching its adolescence: Ontologists' and logicians' health check. IntJMed Inform 2009;78:S86-94.
75. Schulz S, Stenzhorn H, Boeker M. The ontology of biological taxa. Bioinformatics 2008;24:i313.
76. Wolstencroft K, Lord P, Tabernero L, et al. Protein classification using ontology classification. Bioinformatics 2006;22: e530-8.
77. Kohler S, Bauer S, Mungall C, et al. Improving ontologies by automatic reasoning and evaluation of logical definitions. BMC. Bioinformatics 2011;12:418.
78. Kazakov Y, Kroetzsch M, Simancik F. Consequence- Driven Reasoning for Horn SHIQ Ontologies. In Proceedings of the 23rd International Workshop on Description Logics (DL'10). Aachen, Germany: CEUR-WS.org, 2011.
79. Jupp S, Stevens R, Hoehndorf R. Logical Gene Ontology Annotations (GOAL): exploring gene ontology annotations with OWL. J Biomed Semant 2012;3:S3.
80. Tweedie S, Ashburner M, Falls K, et al. FlyBase: enhancing Drosophila Gene Ontology annotations. Nucleic Acids Res 2009;37:D555-9.
81. Sirin E, Parsia B. Pellet: An OWL DL Reasoner. In: Haarslev V, Moeller R, (eds). Proceedings of the 2004 International Workshop on Description Logics, DL2004. Whistler, British Columbia, Canada, 104 CEUR-WS.org, Aachen, Germany, 2004.
82. Noy NF, Shah NH, Whetzel PL, et al. BioPortal: ontologies and integrated data resources at the click of a mouse. Nucleic Acids Res 2009;37:W170-3.
83. Gangemi A. Ontology Design Patterns for Semantic Web Content. In Proceedings of the Fourth International SemanticWeb Conference. Berlin: Springer-Verlag, 2005;262-76.
84. Gkoutos GV, Green EC, Mallon A-MM, et al. Using ontologies to describe mouse phenotypes. Genome Biol 2005;6:R8.
85. Burek P, Hoehndorf R, Loebe F, et al. A top-level ontology of functions and its application in the open biomedical ontologies. Bioinformatics 2006;22:e66-73.
86. Mungall C, Gkoutos G, Smith C, et al. Integrating phenotype ontologies across multiple species. Genome Biol 2010; 11:R2+.
87. Hoehndorf R, Oellrich A, Rebholz-Schuhmann D. Interoperability between phenotype and anatomy ontologies. Bioinformatics 2010;26:3112-8.
88. Searle JR. The Construction of Social Reality. London, UK: Penguin Group, 1995.
89. Wright L. Functions. Philos Rev 1973;82:139-168.
90. Millikan RG. Language, Thought, and Other Biological Categories: New Foundations for Realism. Cambridge, MA, USA: MIT Press, 1988.
91. Guarino N, Welty CA. An overview of OntoClean. In: Staab S, Studer R, (eds). Handbook on Ontologies. Berlin, Germany: Springer Verlag, 2004;151-72.
92. Kutz O, Mossakowski T. A modular consistency proof for DOLCE. In Proceedings of theTwenty-FifthAAAIConference on Artificial Intelligence, 2011.
93. Smith B, Ceusters W, Temmerman R. Wuesteria. Stud HealthTechnol Inform 2005;116:647-52.
94. Boeker M, Tudose I, Hastings J, et al. Unintended consequences of existential quantifications in biomedical ontologies. BMC Bioinformatics 2011;12:456.
95. Schober D, Smith B, Lewis S, et al. Survey-based naming conventions for use in OBO Foundry ontology development. BMC Bioinformatics 2009;10:125.
96. Cohen J. A coefficient of agreement for nominal scales. Educ PsycholMeasure 1960;20:37-46.
97. Artstein R, Poesio M. Inter-coder agreement for computational linguistics. Comput Linguist 2008;34:555-96.
98. Cohen KB, Hunter LE, Palmer M. In Proceedings of the 5th Linguistic AnnotationWorkshop, pp. 82-91. Association for Computational Linguistics, Portland, Oregon, 2011.
99. Jourde J, Manine A-P, Veber P, et al. In Proceedings of the BioNLP SharedTask 2011Workshop, pp. 65-73. Association for Computational Linguistics, Portland, Oregon, 2011.
100. Stevenson M, Guo Y. Disambiguation in the biomedical domain: the role of ambiguity type. J Biomed Inform 2010; 43:972-81.
101. Van Rijsbergen CJ. Information Retrieval. London, UK: Butterworths, 1979.
102. Fawcett T. An introduction to ROC analysis. Pattern Recognit Lett 2006;27:861-874.
103. Xu T, Du L, Zhou Y. Evaluation of GO-based functional similarity measures using S. cerevisiae protein interaction and expression profile data. BMC Bioinformatics 2008;9:472.
104. Huizinga D, Kolawa A. Automated Defect Prevention: Best Practices in Software Management. Hoboken, New Jersey, USA: Wiley-Interscience, 2007.
105. Baumann R, Loebe F, Herre H. Ontology of time in GFO. In Formal Ontology in Information Systems: Proceedings of the Seventh International Conference,. Amsterdam, The Netherlands: IOS Press, 2012;293-306.
106. Camon E, Barrell D, Dimmer E, et al. An evaluation of GO annotation retrieval for BioCreAtIvE and GOA. BMC Bioinformatics 2005;6:S17.
107. Dolan ME, Ni L, Camon E, et al. A procedure for assessing GO annotation consistency. Bioinformatics 2005;21: i136-43.
108. Blond W, Mironov V, Venkatesan A, et al. Reasoning with bio-ontologies: using relational closure rules to enable practical querying. Bioinformatics 2011;27:1562-8.
109. Jacso P. Content evaluation of databases. Annu Rev Inform SciTechnol 1997;32:231-67.
110. Li C, Donizelli M, Rodriguez N, et al. BioModels Database: an enhanced, curated and annotated resource for published quantitative kinetic models. BMC Syst Biol 2010;4:92+.
111. Jansen L, Schulz S. The Ten Commandments of Ontological Engineering. In: Herre H, Hoehndorf R, Loebe F, (eds). OBML 2011Workshop Proceedings. Leipzig, Germany: Markus Loeffler, 2011. p G-11.
112. Stevens R, Malone J, Williams S, et al. Automating generation of textual class definitions from OWL to English. J Biomed Semantics, 2011;2(Suppl 2):S5.
113. Brown S, Chambon P, de Angelis MH, et al. EMPReSS: standardized phenotype screens for functional annotation of the mouse genome. Nat Genet 2005;37:1155.
114. Gkoutos GV, Green EC, Mallon AM, et al. Building mouse phenotype ontologies. Pac Symp Biocomput 2004; 178-89.
115. Hernandez-Boussard T, Whirl-Carrillo M, Hebert JM, et al. The pharmacogenetics and pharmacogenomics knowledge base: accentuating the knowledge. Nucleic Acids Res 2008;36:D913-D918.
116. Chen R, Mias GI, Li-Pook-Than J, et al. Personal omics profiling reveals dynamic molecular and medical phenotypes. Cell 2012;148:1293-307.
117. Samwald M, Coulet A, Huerga I, et al. Semantically enabling pharmacogenomic data for the realization of personalized medicine. Pharmacogenomics 2012;13:201-12.
118. Patel CJ, Bhattacharya J, Butte AJ. An Environment-Wide Association Study (EWAS) on type 2 diabetes mellitus. PLoS One 2010;5:e10746.
119. Barabasi A-LL, Gulbahce N, Loscalzo J. Network medicine: a network-based approach to human disease. NatRev Genet 2011;12:56-68.
120. Gottlieb A, Stein GY, Ruppin E, et al. PREDICT: a method for inferring novel drug indications with application to personalized medicine. Mol Syst Biol 2011;7.
121. Tatonetti NP, Ye PP, Daneshjou R, et al. Data-driven prediction of drug effects and interactions. eng. SciTranslMed 2012;4:125ra31.
122. Washington NL, Haendel MA, Mungall CJ, et al. Linking human diseases to animal models using ontology-based phenotype annotation. PLoS Biol 2009;7:e1000247.
123. Cornet R, de Keizer N. Forty years of SNOMED: a literature review. BMCMed InformDecisMak 2008;8:S2.
124. Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res 2004;32:D267-D270.