A study of terminology auditors' performance for UMLS semantic type assignments

Journal of Biomedical Informatics

Volumn 45, Issue 6, 2012, Pages 1042-1048

  Gu, Huanying Helen ^a   Elhanan, Gai ^b   Perl, Yehoshua ^b   Hripcsak, George ^c   Cimino, James J ^d   Xu, Julia ^d   Chen, Yan ^e   Geller, James ^b   Paul Morrey, C ^f  

^a NEW YORK INSTITUTE OF TECHNOLOGY   (United States)

^b NEW JERSEY INSTITUTE OF TECHNOLOGY   (United States)

^c Columbia University ^*  (United States)

^d NATIONAL INSTITUTES OF HEALTH   (United States)

^e CITY UNIVERSITY OF NEW YORK   (United States)

^f UTAH VALLEY UNIVERSITY   (United States)

Author keywords

Auditing of terminologies; Auditor performance; Auditor reliability; Quality Assurance; Semantic type assignments; UMLS auditing

Indexed keywords

AUDITOR PERFORMANCE; F-MEASURE; HUMAN EXPERT; PILOT STUDIES; SEMANTIC TYPE ASSIGNMENT; SEMANTIC TYPES; UMLS AUDITING;

ERRORS; HEALTH CARE; MANAGEMENT; QUALITY ASSURANCE; TERMINOLOGY;

SEMANTICS;

ACCURACY; ARTICLE; CONCEPT ANALYSIS; ERROR; METRIC SYSTEM; PILOT STUDY; PRIORITY JOURNAL; RECALL; RELIABILITY; SEMANTICS;

HUMANS; REPRODUCIBILITY OF RESULTS; SEMANTICS; TERMINOLOGY AS TOPIC; UNIFIED MEDICAL LANGUAGE SYSTEM;

EID: 84869872304     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2012.05.006     Document Type: Article

References (32)

1. Humphreys B.L., Lindberg D.A.B., Schoolman H.M., Barnett G.O. The Unified Medical Language System: an informatics research collaboration. J Am Med Inform Assoc 1998, 5(1):1-11.
2. Schuyler P.L., Hole W.T., Tuttle M.S., Sherertz D.D. The UMLS Metathesaurus: representing different views of biomedical concepts. Bull Med Libr Assoc 1993, 81(2):217-222.
3. McCray A.T. An upper level ontology for the biomedical domain. Comp Funct Genom 2003, 4:80-84.
4. Gu H., Hripcsak G., Chen Y., Morrey C.P., Elhanan G., Cimino J.J., et al. Evaluation of auditing process of UMLSsemantic typeassignment. Proc AMIA Annu Sympos 2007, 2007:294-298.
5. Chute C.G., Elkin P.L., Fenton S.H., Atkin G.E. A clinical terminology in the post modern era: pragmatic problem list development. Proc AMIA Symp 1998, 795-799.
6. Fung K.W., Hole W.T., Nelson S.J., Srinivasan S., Powell T., Roth L. Integrating SNOMED CT into the UMLS: an exploration of different views of synonymy and quality of editing. J Am Med Inform Assoc. 2005, 12(4):486-494.
7. McCray A.T., Nelson S.J. The representation of meaning in the UMLS. Methods Inf Med 1995, 34:193-201.
8. Chen Y., Perl Y., Geller J., Cimino J.J. Analysis of a study of the users, uses and future agenda of the UMLS. J Am Med Inform Assoc 2007, 14(2):221-231.
9. Peng Y, Halper M, Perl Y, Geller J. Auditing the UMLS for redundant classifications. In: Proceedings of 2002 AMIA annual symposium, San Antonio, TX, November 2002. p. 612-6.
10. Cimino J.J. Auditing the Unified Medical Language System with semantic methods. J Am Med Inform Assoc 1998, 5:41-51.
11. Cimino JJ. Battling Scylla and Charybdis: the search for redundancy and ambiguity in the 2001 UMLS Metathesaurus. In: Overhage JM, editor. Proceedings of 2001 AMIA annual, symposium, 2001. p. 120-4.
12. Bodenreider O. Circular hierarchical relationships in the UMLS: etiology, diagnosis, treatment, complications and prevention. In: Proceedings of AMIA symposium, 2001. p. 57-61.
13. Mougin F, Bodenreider O. Approaches to eliminating cycles in the UMLS Metathesaurus: naive vs. formal. In: Proceedings of 2005 AMIA annual, symposium, 2005. p. 550-4.
14. Halper M, Morrey CP, Chen Y, Elhanan G, Hripcsak G, Perl Y. Auditing hierarchical cycles to locate other inconsistencies in the UMLS. In: Proceedings of AMIA annual symposium; 2011. p. 529-36.
15. Bodenreider O. An object-oriented model for representing semantic locality in the UMLS. In: Proceedings of Medinfo 2001, London, UK, September 2001. p. 161-5.
16. Zhu X., Fan J., Baorto D.M., Weng C., Cimino J.J. A review of auditing methods applied to the content of controlled biomedical terminologies. J Biomed Inform 2009, 42(3):413-425.
17. Gu H., Perl Y., Geller J., Halper M., Liu L., Cimino J.J. Representing the UMLS as an OODB: modeling issues and advantages. J Am Med Inform Assoc 2000, 7(1):66-80. [Selected for reprint in Haux R, Kulikowski C, eds.: Yearbook of Medical Informatics, International Medical Informatics Association, Rotterdam, 2001:271-285].
18. Geller J., Gu H., Perl Y., Halper M. Semantic refinement and error correction in large terminological knowledge bases. Data Knowl Eng 2003, 45(1):1-32.
19. Chen Y., Gu H., Perl Y., Geller J., Halper M. Structural group auditing of a UMLS semantic type's extent. J Biomed Inform 2009, 42(1):41-52.
20. Chen Y., Gu H., Perl Y., Geller J. Structural group-based auditing of missing hierarchical relationships in UMLS. J Biomed Inform 2009, 42(3):452-467.
21. Chen Y., Gu H., Perl Y., Halper M., Xu J. Expanding the extent of a UMLS semantic type via group neighborhood auditing. J Am Med Inform Assoc 2009, 16(5):746-757.
22. Gu H., Perl Y., Elhanan G., Min H., Zhang L., Peng Y. Auditing concept categorizations in the UMLS. Artif Intell Med 2004, 31(1):29-44.
23. Morrey C.P., Geller J., Halper M., Perl Y. The neighborhood auditing tool: a hybrid interface for auditing the UMLS. J Biomed Inform 2009, 42(3):468-489.
24. <>. (accessed 02.15.12). http://web.njit.edu/~kh8/UMLS/SemanticNetwork3.php?2.
25. Dunn G. Design and analysis of reliability studies 1989, Oxford University Press, New York.
26. Hripcsak G., Rothschild A.S. Agreement, the F-measure, and reliability in information retrieval. J Am Med Inform Assoc 2005, 12:296-298.
27. Efron B., Tibshirani R. An Introduction to the Bootstrap 1993, Chapman & Hall, New York.
28. Hripcsak G., Wilcox A. Reference standards, judges, and comparison subjects: roles for experts in evaluating system performance. J Am Med Inform Assoc 2002, 9(1):1-15.
29. Geller J, Morrey CP, Xu J, Halper M, Elhanan G, Hripcsak G. Comparing inconsistent relationship configurations indicating UMLS errors. In: Proc 2009 AMIA Annual Symp; San Francisco, CA; 2009. p. 193-7.
30. Cimino J.J., Min H., Perl Y. Consistency across the hierarchies of the UMLS Semantic Network and Metathesaurus. J Biomed Inform 2003 Dec, 36(6):450-461.
31. Wang Y., Halper M., Wei D., Gu H., Perl Y., Xu J., et al. Auditing complex concepts of SNOMED using a refined hierarchical abstraction network. J Biomed Inform 2012, 45(1):1-14.
32. Wang Y., Halper M., Min H., Perl Y., Chen Y., Spackman K.A. Structural methodologies for auditing SNOMED. J Biomed Inform 2007, 40(5):561-581.