Computer-interpretable clinical guidelines: A methodological review

Journal of Biomedical Informatics

Volumn 46, Issue 4, 2013, Pages 744-763

  Peleg, Mor ^a  

^a UNIVERSITY OF HAIFA   (Israel)

Author keywords

Clinical practice guidelines; Computer interpretable clinical guidelines; Decision support systems; Knowledge representation

Indexed keywords

BIOMEDICAL INFORMATICS; CLINICAL GUIDELINE; CLINICAL PRACTICE GUIDELINES; ELECTRONIC HEALTH RECORD (EHRS); IMPLEMENTATION BARRIERS; SPECIFICATION METHODOLOGIES; TEMPORAL EXPRESSIONS; VALIDATION AND VERIFICATION;

DECISION SUPPORT SYSTEMS; KNOWLEDGE REPRESENTATION;

COMPLIANCE CONTROL;

ANALYTICAL ERROR; COMPUTER; COMPUTER MODEL; DATA EXTRACTION; DECISION SUPPORT SYSTEM; DECISION TREE; DRAWING; ELECTRONIC MEDICAL RECORD; EVOLUTION; HEALTH CARE ORGANIZATION; INTEGRATION; LANGUAGE; LEARNING; LIFE CYCLE; MEDICAL RESEARCH; PATIENT CARE; PATIENT CODING; PRACTICE GUIDELINE; PRIORITY JOURNAL; PUBLICATION; REVIEW; VALIDATION PROCESS; WORKFLOW;

CLINICAL PRACTICE GUIDELINES; COMPUTER-INTERPRETABLE CLINICAL GUIDELINES; DECISION-SUPPORT SYSTEMS; KNOWLEDGE REPRESENTATION;

DECISION MAKING, COMPUTER-ASSISTED; MODELS, THEORETICAL; PRACTICE GUIDELINES AS TOPIC;

EID: 84880326705     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2013.06.009     Document Type: Review

References (115)

1. Institute of Medicine. Clinical practice guidelines we can trust. Consensus report, 2013. <> [accessed on 23.04.13]. http://www.iom.edu/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust.aspx.
2. Peleg M., Tu S.W., Bury J., Ciccarese P., Fox J., Greenes R.A., et al. Comparing computer-interpretable guideline models: a case-study approach. J Am Med Inform Assoc 2003, 10(1):52-68.
3. Latoszek-Berendsen A., Tange H., van den Herik H.J., Hasman A. From clinical practice guidelines to computer-interpretable guidelines. A literature overview. Methods Inf Med 2010, 49(6):550-570.
4. Hripcsak G., Ludemann P., Pryor T.A., Wigertz O.B., Clayton P.D. Rationale for the Arden syntax. Comput Biomed Res 1994, 27(4):291-324.
5. Koutkias V., Kilintzis V., Stalidis G., Lazou K., Niès J., Durand-Texte L., et al. Knowledge engineering for adverse drug event prevention: on the design and development of a uniform, contextualized and sustainable knowledge-based framework. J Biomed Inform 2012, 45(3):495-506.
6. Serban R., ten Teije A., van Harmelen F., Marcos M., Polo-Conde C. Extraction and use of linguistic patterns for modelling medical guidelines. Artif Intell Med 2007, 39(2):137-149.
7. Shiffman R.N., Karras B.T., Agrawal A., Chen R., Marenco L., Nath S. GEM: a proposal for a more comprehensive guideline document model using XML. J Am Med Inform Assoc 2000, 7(5):488-498.
8. Georg G., Seroussi B., Bouaud J. Extending the GEM model to support knowledge extraction from textual guidelines. Int J Med Inform 2005, 74(2-4):79-87.
9. Colombet I., Aguirre-Junco A.-R., Zunino S., Jaulent M.-C., Leneveut L., Chatellier G. Electronic implementation of guidelines in the EsPeR system: a knowledge specification method. Int J Med Inform 2005, 74(7-8):597-604.
10. Seroussi B., Bouaud J., Antoine E.-C. ONCODOC: a successful experiment of computer-supported guideline development and implementation in the treatment of breast cancer. Artif Intell Med 2001, 22(1):43-64.
11. Seroussi B., Bouaud J., Chatellier G. Guideline-based modeling of therapeutic strategies in the special case of chronic diseases. Intl J Med Inform 2005, 74(2-4):89-99.
12. Sanders G.D., Nease R.F., Owens D.K. Design and pilot evaluation of a system to develop computer-based site-specific practice guidelines from decision models. Med Decis Mak 2000, 20(2):145-159.
13. Papageorgiou E.I., Roo J.D., Huszka C., Colaert D. Formalization of treatment guidelines using Fuzzy Cognitive Maps and semantic web tools. J Biomed Inform 2012, 45(1):45-60.
14. Shahar Y. The "Human Cli-Knowme" project: building a universal, formal, procedural and declarative clinical knowledge base, for the automation of therapy and research. Lect Not Artif Intell 6924 2012, 1-22. Springer.
15. Musen M.A., Tu S.W., Das A.K., Shahar Y. EON: a component-based approach to automation of protocol-directed therapy. J Am Med Inform Assoc 1996, 3(6):367-388.
16. Shahar Y., Miksch S., Johnson P. The Asgaard project: a task-specific framework for the application and critiquing of time-oriented clinical guidelines. Artif Intell Med 1998, 14(1-2):29-51.
17. Seyfang A., Miksch S., Marcos M. Combining diagnosis and treatment using Asbru. Int J Med Inform 2002, 68(1-3):49-57.
18. Latoszek-Berendsen A., Talmon J., de Clercq P., Hasman A. With good intentions. Int J Med Inform 2007, 76(Suppl. 3):S440-S446.
19. Latoszek-Berendsen A., de Clercq P., van den Herik J., Hasman A. Intention-based expressions in GASTINE. Methods Inform Med 2009, 48(4):391-396.
20. de-Clercq P.A., Blom J.A., Korsten H.H., Hasman A. Approaches for creating computer-interpretable guidelines that facilitate decision support. Artif Intell Med 2004, 31(1):1-27.
21. Anselma L., Terenziani P., Montani S., Bottrighi A. Towards a comprehensive treatment of repetitions, periodicity and temporal constraints in clinical guidelines. Artif Intell Med 2006, 38(2):171-195.
22. Terenziani P., Molino G., Torchio M. A modular approach for representing and executing clinical guidelines. Artif Intell Med 2001, 23(3):249-276.
23. Boxwala A.A., Tu S., Zeng Q., Peleg M., Ogunyemi O., Greenes R.A., et al. Towards a representation format for sharable clinical guidelines. J Biomed Inform 2001, 34:157-169.
24. Peleg M., Boxwala A.A., Bernstam E., Tu S., Greenes R.A., Shortliffe E.H. Sharable representation of clinical guidelines in GLIF: relationship to the Arden syntax. J Biomed Inform 2001, 34(3):170-181.
25. Boxwala A.A., Peleg M., Tu S., Ogunyemi O., Zeng Q., Wang D., et al. GLIF3: a representation format for sharable computer-interpretable clinical practice guidelines. J Biomed Inform 2004, 37(3):147-161.
26. Skonetzki A., Gausepohl H.J., van der Haak M., Knaebel S., Linderkamp O., Wetter T. HELEN, a modular framework for representing and implementing clinical practice guidelines. Methods Inf Med 2004, 43(4):413-426.
27. Panzarasa S., Madde S., Quaglini S., Pistarini C., Stefanelli M. Evidence-based careflow management systems: the case of post-stroke rehabilitation. J Biomed Inform 2002, 35(2):123-139.
28. Quaglini S., Stefanelli M., Lanzola G., Caporusso V., Panzarasa S. Flexible guideline-based patient careflow systems. Artif Intell Med 2001, 22:65-80.
29. Ciccarese P., Caffi E., Quaglini S., Stefanelli M. Architectures and tools for innovative health information systems: the guide project. Int J Med Inform 2005, 74(7-8):553-562.
30. Sutton D.R., Fox J. The syntax and semantics of the PROforma guideline modeling language. J Am Med Inform Assoc 2003, 10(5):433-443.
31. Tu S.W., Campbell J.R., Glasgow J., Nyman M.A., McClure R., McClay J., et al. The SAGE guideline model : achievements and overview. J Am Med Inform Assoc 2007, 14(5):589-598.
32. Schadow G., Russler D.C., McDonald C.J. Conceptual alignment of electronic health record data with guideline and workflow knowledge. Intl J Med Inform 2001, 64(2-3):259-274.
33. Sordo M, Boxwala AA, Ogunyemi O, Greenes RA. Description and status update on GELLO: a proposed standardized object-oriented expression language for clinical decision support. In: Medinfo, 2004. p. 164-8.
34. Patel V.L., Arocha J.F., Diermeier M., Shortliffe E.H., Greenes R.A. Methods of cognitive analysis to support the design and evaluation of biomedical systems: the case of clinical practice guidelines. J Biomed Inform 2001, 34(1):52-66.
35. Peleg M., Gutnik L.A., Snow V., Patel V.L. Interpreting procedures from descriptive guidelines. J Biomed Inform 2006, 39(2):184-195.
36. Stausberg J., Bilir H., Waydhas C., Ruchholtz S. Guideline validation in multiple trauma care through business process modeling. Int J Med Inform 2003, 70(2-3):301-307.
37. Patel V.L., Allen V.G., Arocha J.F., Shortliffe E.H. Representing a clinical guideline in GLIF: individual and collaborative expertise. J Am Med Inform Assoc 1998, 5(5):467-483.
38. Ohno-Machado L., Gennari J.H., Murphy S., Jain N.L., Tu S.W., Oliver D.E., et al. The guideline interchange format: a model for representing guidelines. J Am Med Inform Assoc 1998, 5(4):357-372.
39. Patel V.L., Branch T., Wang D., Peleg M., Boxwala A.A. Analysis of the process of encoding guidelines: a comparison of GLIF2 and GLIF3. Methods Inform Med 2002, 41(2):102-113.
40. Shalom E., Bar G., Yarkoni A., Young O., Martins S.B., Vaszar L., et al. A quantitative assessment of a methodology for collaborative specification and evaluation of clinical guidelines. J Biomed Inform 2008, 41(6):889-903.
41. Shiffman R.N., Michel G., Rosenfeld R.M., Davidson C. Building better guidelines with BRIDGE-Wiz: development and evaluation of a software assistant to promote clarity, transparency, and implementability. J Am Med Inform Assoc 2012, 19(1):94-101.
42. Shiffman R.N., Dixon J., Brandt C., Essaihi A., Hsiao A., Michel G., et al. The guideline implementability appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation. BMC Med Inform Decis Mak 2005, 5:23.
43. Shiffman R.N., Michel G., Essaihi A., Thornquist E. Bridging the guideline implementation gap: a systematic, document-centered approach to guideline implementation. J Am Med Inform Assoc 2004, 11(5):418-426.
44. Svátek V., Růzicka M. Step-by-step mark-up of medical guideline documents. Intl J Med Inform 2003, 70(2-3):329-335.
45. Peleg M., Tu S.W. Design patterns for clinical guidelines. Artif Intel Med 2009, 47(1):1-24.
46. Serban R., ten Teije A. Exploiting thesauri knowledge in medical guideline formalization. Methods Inform Med 2009, 48(5):468-474.
47. Kaiser K., Miksch S. Versioning computer-interpretable guidelines: semi-automatic modeling of "living guidelines" using an information extraction method. Artif Intell Med 2009, 46(1):55-66.
48. Kaiser K., Akkayaa C., Miksch S. How can information extraction ease formalizing treatment processes in clinical practice guidelines? A method and its evaluation. Artif Intell Med 2010, 39(2):151-163.
49. Kosara R., Miksch S. Metaphors of movement: a visualization and user interface for time-oriented, skeletal plans. Artif Intell Med 2001, 22(2):11-131.
50. Goud R., Hasman A., Strijbis A.-M., Peek N. A parallel guideline development and formalization strategy to improve the quality of clinical practice guidelines. Int J Med Inform 2009, 78(8):513-520.
51. Bates D.W., Kuperman G.J., Wang S., Gandhi T., Kittler A., Volk L., et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003, 10(6):523-530.
52. Peleg M., Keren S., Denekamp Y. Mapping computerized clinical guidelines to electronic medical records: knowledge-data ontological mapper (KDOM). J Biomed Inform 2008, 41(1):180-201.
53. Johnson P.D., Tu S.W., Musen M.A., Purves I. A virtual medical record for guideline-based decision support. Proc AMIA Symp 2001, 294-298.
54. German E., Leibowitz A., Shahar Y. An architecture for linking medical decision-support applications to clinical databases and its evaluation. J Biomed Inform 2009, 42(2):203-218.
55. González-Ferrer A., ten Teije A., Fdez-Olivares J., Milian K. Automated generation of patient-tailored electronic care pathways by translating computer-interpretable guidelines into hierarchical task networks. Artif Intell Med. 2013, 57(2):91-109.
56. Boehm B.W. Software risk management 1989, IEEE Computer Society Press.
57. Peleg M., Shachak A., Wang D., Karnieli E. Using multi-perspective methodologies to study user interactions with the front-end of a guideline-based decision-support system for diabetic-foot care. Intl J Med Inform 2009, 78(7):482-493.
58. Miller P.L., Frawley S.J., Sayward F.G. Maintaining and incrementally revalidating a computer-based clinical guideline: a case study. J Biomed Inform. 2001, 34(2):99-111.
59. Choi J., Currie L.M., Wang D., Bakken S. Encoding a clinical practice guideline using guideline interchange format: a case study of a depression screening and management guideline. Intl J Med Inform 2007, 76(Suppl. 2). S302-7.
60. Miller P.L. Domain-constrained generation of clinical condition sets to help test computer-based clinical guidelines. J Am Med Inform Assoc 2001, 8(2):131-145.
61. Bottrighi A, Chesani F, Mello P, Montali M, Montani S, Terenziani P. Conformance checking of executed clinical guidelines in presence of basic medical knowledge. In: Business process management workshops, 2011. p. 200-11.
62. Duftschmid G., Miksch S. Knowledge-based verification of clinical guidelines by detection of anomalies. Artif Intell Med 2001, 22(1):23-41.
63. Duftschmid G., Miksch S., Gall W. Verification of temporal scheduling constraints in clinical practice guidelines. Artif Intell Med 2002, 25(2):93-121.
64. Pérez B., Porres I. Authoring and verification of clinical guidelines: a model driven approach. J Biomed Inform 2010, 43(4):520-536.
65. Domínguez E., Pérez B., Zapata M. Towards a traceable clinical guidelines application. A model-driven approach. Methods Inform Med 2010, 49(6):571-580.
66. Hommersom A., Groot P., Balser M., Lucas P. Chapter 4. Formal methods for verification of clinical practice guidelines. Studies in health technology and informatics 2008, vol. 139:63-80. IOS Press. A. ten Teije, S. Miksch, P.J.F. Lucas (Eds.).
67. Bottrighi A., Giordano L., Molino G., Montani S., Terenziani P., Torchio M. Adopting model checking techniques for clinical guidelines verification. Artif Intell in Med 2010, 48(1):1-19.
68. ten Teije A., Marcos M., Balser M., van Croonenborg J., Duelli C., van Harmelen F., et al. Improving medical protocols by formal methods. Artif Intell Med 2006, 36(3):193-209.
69. Wilk S., Michalowski W., Michalowski M., Farion K., Hing M.M., Mohapatra S. Mitigation of adverse interactions in pairs of clinical practice guidelines using constraint logic programming. J Biomed Inform 2013, 46(2):341-353.
70. Grando M.A., Glasspool D., Fox J. A formal approach to the analysis of clinical computer-interpretable guideline modeling languages. Artif Intell Med 2012, 54(1):1-13.
71. Mulyar N., van der Aalst W.M.P., Peleg M. A pattern-based analysis of clinical computer-interpretable guideline modeling languages. J Am Med Inform Assoc 2007, 14(6):781-787.
72. Isern D., Moreno A. Computer-based execution of clinical guidelines: a review. Int J Med Inform 2008, 77(12):787-808.
73. Wang D., Peleg M., Tu S.W., Boxwala A.A., Ogunyemi O., Zeng Q., et al. Design and implementation of the GLIF3 guideline execution engine. J Biomed Inform 2004, 37(5):305-318.
74. Wang D, Peleg M, Bu D, Cantor M, Landesberg G, Lunenfeld E, et al. GESDOR - a generic execution model for sharing of computer-interpretable clinical practice guidelines. In: Proc AMIA Symp, 2003. p. 694-8.
75. Young O., Shahar Y., Liel Y., Lunenfeld E., Bar G., Shalom E., et al. Runtime application of Hybrid-Asbru clinical guidelines. J Biomed Inform 2007, 40(5):507-526.
76. de Clercq P.A., Hasman A., Blom J.A., Korsten H.H. The application of ontologies and problem-solving methods for the development of shareable guidelines. Artif Intell Med 2001, 22(1):1-22.
77. de-Clercq P.A., Hasman A., Blom J.A., Korsten H.H. Design and implementation of a framework to support the development of clinical guidelines. Int J Med Inform 2001, 64(2-3):285-318.
78. Shahar Y., Young O., Shalom E., Galperin M., Mayaffit A., Moskovitch R., et al. A framework for a distributed, hybrid, multiple-ontology clinical guideline library, and automated guideline-support tools. J Biomed Inform 2004, 37(5):325-344.
79. Moskovitch R., Shahar Y. Vaidurya: a multiple-ontology, concept-based, context-sensitive clinical-guideline search engine. J Biomed Inform 2009, 42(1):11-21.
80. Moskovitch R., Cohen-Kashi S., Dror U., Levy I., Maimon A., Shahar Y. Multiple hierarchical classification of free-text clinical guidelines. Artif Intell Med 2006, 37(3):177-190.
81. Grando A., Peleg M., Glasspool D. A goal-oriented framework for specifying clinical guidelines and handling medical errors. J Biomed Inform 2009, 43(2):287-299.
82. Grando A.M., Peleg M., Cuggia M., Glasspool D. Patterns for collaborative work in health care teams. Artif Intell Med 2011, 53(3):139-160.
83. Grando M.A., Glasspool D., Boxwala A. Argumentation logic for the flexible enactment of goal-based medical guidelines. J Biomed Inform 2012, 45(5):938-949.
84. Staroselsky M., Volk La, Tsurikova R., Pizziferri L., Lippincott M., Wald J., et al. Improving electronic health record (EHR) accuracy and increasing compliance with health maintenance clinical guidelines through patient access and input. Int J Med Inform 2005, 75(10-11):693-700.
85. van de Klundert J., Gorissen P., Zeemering S. Measuring clinical pathway adherence. J Biomed Inform 2010, 43(6):861-872.
86. Groot P., Hommersom A., Lucas P.J., Merk R.J., ten Teije A., van Harmelen F., et al. Using model checking for critiquing based on clinical guidelines. Artif Intell Med 2009, 46(1):19-36.
87. Advani A, Shahar Y, Musen M. Medical quality assessment by scoring adherence to guideline intentions. In: AMIA symp, 2001. p. 2-6.
88. Quaglini S. Chapter 9. Compliance with Clinical Practice Guidelines. Studies in health technology and informatics 2008, vol. 139:160-179. IOS Press. A. ten Teije, S. Miksch, P.J.F. Lucas (Eds.).
89. van der Aalst W.M.P. Process mining: discovery, conformance and enhancement of business processes 2011, Springer.
90. Lin F., Chou S., Pan S., Chen Y. Mining time dependency patterns in clinical pathways. Int J Med Inform 2001, 62(1):11-25.
91. Huang Z., Lu X., Duan H. On mining clinical pathway patterns from medical behaviors. Artif Intell Med 2012, 56(1):35-50.
92. Huang Z., Lu X., Duan H., Fan W. Summarizing clinical pathways from event logs. J Biomed Inform 2012, 46(1):111-127.
93. Aigner W., Miksch S. CareVis: integrated visualization of computerized protocols and temporal patient data. Artif Intell Med 2006, 37(3):203-218.
94. Grandi F., Mandreoli F., Martoglia R. Efficient management of multi-version clinical guidelines. J Biomed Inform 2012, 45(6):1120-1136.
95. Boxwala A., Rocha B.H., Maviglia S., Kashyap V., Meltzer S., Kim J., et al. A multi-layered framework for disseminating knowledge for computer-based decision support. J Am Med Inform Assoc 2011, 18(Suppl. 1). i132-9.
96. Healthcare information technology standards panel. C 32 - HITSP summary documents using HL7 continuity of care document (CCD) component. <>http://www.hitsp.org/ConstructSet_Details.aspx?&PrefixAlpha=4&PrefixNumeric=32.
97. Kawamoto K., Hongsermeier T., Wright A., Lewis J., Bell D.S., Middleton B. Key principles for a national clinical decision support knowledge sharing framework: synthesis of insights from leading subject matter experts. J Am Med Inform Assoc 2013, 20(1):199-207.
98. Morningside Group. Collaborative development of a knowledge management repository to accelerate adoption of clinical decision support, 2007. <>http://www.amia.org/public-policy/reports-and-fact-sheets/morningside-initiative-concept-paper.
99. Gooch P., Roudsari A. Computerization of workflows, guidelines, and care pathways: a review of implementation challenges for process-oriented health information systems. J Am Med Inform Assoc 2011, 18(6):738-748.
100. Wang D., Peleg M., Tu S.W., Boxwal a AA., Greenes R.A., Patel V.L., et al. Representation primitives, process models and patient data in computer-interpretable clinical practice guidelines: a literature review of guideline representation models. Intl J Med Inform. 2002, 68(1-3):59-70.
101. Heller B, Löffler M, Musen M, Stefanelli M, editors. Computer-Based Support for Clinical Guidelines and Protocols Editors. In: Studies in health technology and informatics, vol. 83, 2001.
102. Kaiser K, Miksch S, Tu SW, editors. Computer-based support for clinical guidelines and protocols. In: Proceedings of the symposium on computerized guidelines and protocols (CPG 2004). Studies in health technology and informatics, vol. 101, 2004.
103. ten Teije A., Miksch S., Lucas P. Computer-based medical guidelines and protocols: a primer and current trends. Studies in health technology and informatics 2008, vol. 139. IOS Press.
104. Riaño D., Real F., López-Vallverdú J.A., Campana F., Ercolani S., Mecocci P., et al. An ontology-based personalization of health-care knowledge to support clinical decisions for chronically ill patients. J Biomed Inform 2012, 45(3):429-446.
105. Fux A, Peleg M, Soffer P. How can personal context improve adherence to clinical guidelines. In: Prohealth/KR4HC. Tallinn, Estonia, September 3, 2012.
106. Kawamoto K., Honey A., Rubin K. The HL7-OMG healthcare services specification project: motivation, methodology, and deliverables for enabling a semantically interoperable service-oriented architecture for healthcare. J Am Med Inform Assoc 2009, 16(6):874-881.
107. Soffer P, Ghattas J, Peleg M. A goal-based approach for learning in business processes. In: Salinesi C, Souveyet C, Ralyte J, editors. Intentional perspectives on information systems engineering. Springer; 2010. p. 239-56. ISBN-10: 3642125433 ISBN-13: 978-3642125430.
108. Ghattas J., Peleg M., Soffer P. Data requirements for process learning. Int J Knowl Org 2013, 3(1). [special issue on Process support in healthcare - Part II].
109. Reichert M., Dadam P. ADEPT flex - supporting dynamic changes of workflows without losing control. J Intell Inform Sys. 1997, 10(2):93-129.
110. Marcos M, Maldonado JA, Moner D, Bosc D. An archetype-based solution for the interoperability of computerised guidelines and electronic health records. In: AIME. 2011. p. 276-85.
111. Le XH, Luque A, Wang D. Development of guideline-driven mobile applications for clinical education and decision support with customization to individual patient cases. In: Proc AMIA annual symp, 2012. p. 14642.
112. Jafarpour B, Abidi SR, Sibte S, Abidi R. Exploiting OWL reasoning services to execute ontologically-modeled clinical practice guidelines. In: AIME. LNCS, vol. 6747; 2011. p. 307-11.
113. Leonardi G, Bottrighi A, Galliani G, Terenziani P, Messina A, Francesco Della Corte. Exceptions handling within GLARE clinical guideline framework. In: Proceedings of AMIA, symposium, 2012. p. 512-21.
114. Mans R, Schonenberg H, Leonardi G, Panzarasa S, Cavallini A, Quaglini S, et al. Process mining techniques: an application to stroke care. In: Medical informatics, Europe, 2008. p. 573-8.
115. Anselma L., Bottrighi A., Montani S., Terenziani P. Managing proposals and evaluations of updates to medical knowledge: theory and applications. J Biomed Inform 2013, 46(2):363-376.