AskHERMES: An online question answering system for complex clinical questions

Journal of Biomedical Informatics

Volumn 44, Issue 2, 2011, Pages 277-288

  Cao, YongGang ^a,g   Liu, Feifan ^a   Simpson, Pippa ^b   Antieau, Lamont ^a   Bennett, Andrew ^b,c   Cimino, James J ^d   Ely, John ^e   Yu, Hong ^a,f  

^a UNIVERSITY OF WISCONSIN MILWAUKEE   (United States)

^b MEDICAL COLLEGE OF WISCONSIN   (United States)

^c VETERANS AFFAIRS MEDICAL CENTER   (United States)

^d NATIONAL INSTITUTES OF HEALTH   (United States)

^e UNIVERSITY OF IOWA HOSPITALS AND CLINICS   (United States)

^f University of Wisconsin Milwaukee   (United States)

^g AMAZON   (United States)

Author keywords

Answer presentation; Clinical question answering; Passage retrieval; Question analysis; Summarization

Indexed keywords

ANSWER PRESENTATION; PASSAGE RETRIEVAL; QUESTION ANALYSIS; QUESTION ANSWERING; SUMMARIZATION;

NATURAL LANGUAGE PROCESSING SYSTEMS; QUERY PROCESSING; SEMANTICS;

SEARCH ENGINES;

ACUTE CORONARY SYNDROME; ARTICLE; CLINICAL TRIAL (TOPIC); DATA EXTRACTION; DIABETES MELLITUS; EVIDENCE BASED PRACTICE; HEALTH CARE QUALITY; HUMAN; INFORMATION RETRIEVAL; INFORMATION SERVICE; INTERNET; MACHINE LEARNING; MEDICAL INFORMATION; ONLINE ANALYTICAL PROCESSING; PHYSICIAN; PRIORITY JOURNAL; SEMANTICS;

ALGORITHMS; CLINICAL MEDICINE; DATABASES, FACTUAL; EXPERT SYSTEMS; INFORMATION STORAGE AND RETRIEVAL; NATURAL LANGUAGE PROCESSING; ONLINE SYSTEMS; SOFTWARE;

EID: 79952770126     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2011.01.004     Document Type: Article

References (106)

1. Timpka T., Arborelius E. The GP's dilemmas: a study of knowledge need and use during health care consultations. Methods Inform Med 1990, 29:23-29.
2. Bergus G.R., Randall C.S., Sinift S.D., Rosenthal D.M. Does the structure of clinical questions affect the outcome of curbside consultations with specialty colleagues?. Arch Family Med 2000, 9:541-547.
3. Ely J.W., Burch R.J., Vinson D.C. The information needs of family physicians: case-specific clinical questions. J Family Pract 1992, 35:265-269.
4. Osheroff J.A., Forsythe D.E., Buchanan B.G., Bankowitz R.A., Blumenfeld B.H., Miller R.A. Physicians' information needs: analysis of questions posed during clinical teaching. Ann Int Med 1991, 114:576-581.
5. Covell D.G., Uman G.C., Manning P.R. Information needs in office practice: are they being met?. Ann Int Med 1985, 103:596-599.
6. Smith R. What clinical information do doctors need?. BMJ 1996, 313:1062-1068.
7. Ely J.W., Osheroff J.A., Chambliss M.L., Ebell M.H., Rosenbaum M.E. Answering physicians' clinical questions: obstacles and potential solutions. J Am Med Inform Assoc 2005, 12:217-224.
8. Cimino JJ, Li J, Graham M, Currie LM, Allen M, Bakken S, et al. Use of online resources while using a clinical information system. In: AMIA ann symp proc; 2003. p. 175-9.
9. Fletcher R.H., Fletcher S.W. Evidence-based approach to the medical literature. J Gen Int Med 1997, 12(Suppl. 2):S5-S14.
10. Hersh W.R., Crabtree M.K., Hickam D.H., Sacherek L., Friedman C.P., Tidmarsh P., et al. Factors associated with success in searching MEDLINE and applying evidence to answer clinical questions. J Am Med Inform Assoc 2002, 9:283-293.
11. Cullen R.J. In search of evidence: family practitioners' use of the Internet for clinical information. J Med Lib Assoc 2002, 90:370-379.
12. Stephens M.B., Von Thun A.M. Military medical informatics: accessing information in the deployed environment. Military Med 2009, 174:259-264.
13. Kitchin D.R., Applegate K.E. Learning radiology a survey investigating radiology resident use of textbooks, journals, and the internet. Acad Radiol 2007, 14:1113-1120.
14. Tang H., Ng J.H. Googling for a diagnosis - use of Google as a diagnostic aid: internet based study. BMJ 2006, 333:1143-1145.
15. Purcell G.P., Wilson P., Delamothe T. The quality of health information on the internet. BMJ 2002, 324:557-558.
16. Jadad A.R., Gagliardi A. Rating health information on the Internet: navigating to knowledge or to Babel?. JAMA 1998, 279:611-614.
17. Silberg W.M., Lundberg G.D., Musacchio R.A. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor - let the reader and viewer beware. JAMA 1997, 277:1244-1245.
18. Glennie E., Kirby A. The career of radiography: information on the web. J Diag Radiogr Imaging 2006, 6:25-33.
19. Childs S. Judging the quality of internet-based health information. Perform Meas Metrics 2005, 6:80-96.
20. Griffiths K.M., Tang T.T., Hawking D., Christensen H. Automated assessment of the quality of depression websites. J Med Int Res 2005, 7:e59.
21. Griffiths K.M., Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ 2000, 321:1511-1515.
22. Wyatt J.C. Commentary: measuring quality and impact of the World Wide Web. BMJ 1997, 314:1879-1881.
23. Benigeri M., Pluye P. Shortcomings of health information on the Internet. Health Promot Int 2003, 18:381-386.
24. Cline R.J., Haynes K.M. Consumer health information seeking on the Internet: the state of the art. Health Educ Res 2001, 16:671-692.
25. Freeman M.K., Lauderdale S.A., Kendrach M.G., Woolley T.W. Google Scholar versus PubMed in locating primary literature to answer drug-related questions. Ann Pharmacother 2009, 43:478-484.
26. Ely J.W., Osheroff J.A., Ebell M.H., Bergus G.R., Levy B.T., Chambliss M.L. Evans ER: analysis of questions asked by family doctors regarding patient care. BMJ 1999, 319:358-361.
27. De Leo G., LeRouge C., Ceriani C., Niederman F. Websites most frequently used by physician for gathering medical information. AMIA Annu Symp Proc 2006, 902.
28. McCord G., Smucker W.D., Selius B.A., Hannan S., Davidson E., Schrop S.L., et al. Answering questions at the point of care: do residents practice EBM or manage information sources?. Acad Med 2007, 82:298-303.
29. Goodyear-Smith F., Kerse N., Warren J., Arroll B. Evaluation of e-textbooks. DynaMed, MD Consult and UpToDate. Aust Family Phys 2008, 37:878-882.
30. Phua J., Lim T.K. How residents and interns utilise and perceive the personal digital assistant and UpToDate. BMC Med Educ 2008, 8:39.
31. Cimino J.J., Borotsov D.V. Leading a horse to water: using automated reminders to increase use of online decision support. AMIA Annu Symp Proc 2008, 116-120.
32. Hoogendam A., Stalenhoef A.F., Robbe P.F., Overbeke A.J. Answers to questions posed during daily patient care are more likely to be answered by UpToDate than PubMed. J Med Int Res 2008, 10:e29.
33. Yu H, Sable C, Zhu HR. Classifying medical questions based on an evidence taxonomy. In: Proceedings of the AAAI 2005 workshop on question answering in restricted domains; 2005.
34. Yu H, Sable C. Being Erlang Shen. Identifying answerable questions. In: Proceedings of the nineteenth international joint conference on artificial intelligence on knowledge and reasoning for answering questions; 2005.
35. Yu H. Towards answering biological questions with experimental evidence. Automatically identifying text that summarize image content in full-text articles. AMIA Annu Symp Proc 2006, 834-838.
36. Lee M., Cimino J., Zhu H.R., Sable C., Shanker V., Ely J., et al. Beyond information retrieval-medical question answering. AMIA Annu Symp Proc 2006, 469-473.
37. Lee M., Wang W., Yu H. Exploring supervised and unsupervised methods to detect topics in biomedical text. BMC Bioinformatics 2006, 7:140.
38. Yu H., Kaufman K. A cognitive evaluation of four online search engines for answering definitional questions posed by physicians. Pacific Symp Biocomput 2007, 12:328-339.
39. Yu H., Lee M., Kaufman D., Ely J., Osheroff J., Hripcsak G., et al. Development, implementation, and a cognitive evaluation of a definitional question answering system for physicians. J Biomed Inform 2007, 40:236-251.
40. Yu H., Cao Y.G. Automatically extracting information needs from ad hoc clinical questions. AMIA Annu Symp Proc 2008, 96-100.
41. Yu H, Cao YG. Using the weighted keyword models to improve biomedical information retrieval. In: AMIA summit on translational bioinformatics, San Francisco, USA; 2009.
42. Cao Y.G., Ely J., Antieau L., Yu H. Evaluation of the clinical question answering presentation. BioNLP 2009.
43. Yu H, Cao YG. Using the weighted keyword models to improve clinical question answering. In: IEEE international conference on bioinformatics & biomedicine workshop NLP approaches for unmet information needs in health car; 2009.
44. Hersh W, Cohen A, Ruslen L, Roberts P. TREC 2007 genomics track overview. In: The TREC genomics track conference; 2007.
45. Hersh W, Cohen A, Roberts P, Rekapalli H. TREC 2006 genomics track overview. In: TREC genomics track conference; 2006.
46. Voorhees EM. The TREC-8 question answering track report. In: Proceedings of TREC; 1999.
47. Hersh W. The quality of information on the World Wide Web. J Am Coll Dent 1999, 66:43-45.
48. Hearst M, Pedersen J. Reexamining the cluster hypothesis: scatter/gather on retrieval results. In: The 19th annual international ACM conference on research and development in information retrieval (SIGIR-96); 1996.
49. Pratt W. Dynamic organization of search results using the UMLS. Proc AMIA Annu Fall Symp 1997, 480-484.
50. Cimino J.J., Aguirre A., Johnson S.B., Peng P. Generic queries for meeting clinical information needs. Bull Med Lib Assoc 1993, 81:195-206.
51. Zweigenbaum P. Question answering in biomedicine. In: EACL workshop on natural language processing for question answering, Budapest; 2003. p. 1-4.
52. Zweigenbaum P. Question-answering for biomedicine: methods and state of the art. In: MIE 2005 workshop; 2005.
53. Niu Y, Hirst G. Analysis of semantic classes in medical text for question answering. In: ACL 2004 workshop on question answering in restricted domains; 2004.
54. Niu Y, Hirst G, McArthur G, Rodriguez-Gianolli P. Answering clinical questions with role identification. In: ACL workshop on natural language processing in biomedicine; 2003.
55. Del Fiol G., Haug P.J., Cimino J.J., Narus S.P., Norlin C., Mitchell J.A. Effectiveness of topic-specific infobuttons: a randomized controlled trial. J Am Med Inform Assoc 2008, 15:752-759.
56. Collins S.A., Currie L.M., Bakken S., Cimino J.J. Information needs, infobutton manager use, and satisfaction by clinician type: a case study. J Am Med Inform Assoc 2008.
57. Cimino J. Infobuttons: anticipatory passive decision support. AMIA Annu Symp Proc 2008, 1203-1204.
58. Cimino J.J. Use, usability, usefulness, and impact of an infobutton manager. AMIA Annu Symp Proc 2006, 151-155.
59. Lei J., Chen E.S., Stetson P.D., McKnight L.K., Mendonca E.A., Cimino J.J. Development of infobuttons in a wireless environment. AMIA Annu Symp Proc 2003, 906.
60. Cimino J.J., Li J. Sharing infobuttons to resolve clinicians' information needs. AMIA Annu Symp Proc 2003, 815.
61. Cimino J.J., Li J., Bakken S., Patel V.L. Theoretical, empirical and practical approaches to resolving the unmet information needs of clinical information system users. Proc AMIA Symp 2002, 170-174.
62. Cimino J.J., Elhanan G., Zeng Q. Supporting infobuttons with terminological knowledge. Proc AMIA Annu Fall Symp 1997, 528-532.
63. Mendonça EA, Kaufman D, Johnson SB. Answering information needs in workflow. In: Proceedings of the 9th world congress on health information and libraries; 2005.
64. Chase H.S., Kaufman D.R., Johnson S.B., Mendonca E.A. Voice capture of medical residents' clinical information needs during an inpatient rotation. J Am Med Inform Assoc 2009, 16:387-394.
65. Elhadad N., Kan M., Klavans J.L., McKeown K.R. Customization in a unified framework for summarizing medical literature. Artif Intell Med 2005, 33:179-198.
66. Elhadad N., McKeown K., Kaufman D., Jordan D. Facilitating physicians' access to information via tailored text summarization. AMIA Annu Symp Proc 2005, 226-230.
67. McKeown K, Chang SF, Cimino JJ, Feiner SK, Friedman C, Gravano L, et al. PERSIVAL, a system for personalized search and summarization over multimedia healthcare information. In: Proceedings of the 1st ACM/IEEE-CS joint conference on digital libraries; 2001.
68. Ely J.W., Osheroff J.A., Ferguson K.J., Chambliss M.L., Vinson D.C., Moore J.L. Lifelong self-directed learning using a computer database of clinical questions. J Family Pract 1997, 45:382-388.
69. D'Alessandro D.M., Kreiter C.D., Peterson M.W. An evaluation of information-seeking behaviors of general pediatricians. Pediatrics 2004, 113:64-69.
70. Ely J.W., Osheroff J.A., Gorman P.N., Ebell M.H., Chambliss M.L., Pifer E.A., et al. A taxonomy of generic clinical questions: classification study. BMJ 2000, 321:429-432.
71. Seol Y.H., Kaufman D.R., Mendonca E.A., Cimino J.J., Johnson S.B. Scenario-based assessment of physicians' information needs. Medinfo 2004, 11:306-310.
72. Demner-Fushman D., Lin J. Answering clinical questions with knowledge-based and statistical techniques. Comput Linguist 2007, 33:63-103.
73. Huang X., Lin J., Demner-Fushman D. Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annu Symp Proc 2006, 359-363.
74. Demner-Fushman D, Lin J. Answer extraction, semantic clustering, and extractive summarization for clinical question answering. In: Proceedings of the 21st international conference on computational linguistics and 44th annual meeting of the association for computational linguistics (COLING/ACL); 2006. p. 945-52.
75. Lin J, Demner-Fushman D. The role of knowledge in conceptual retrieval: a study in the domain of clinical medicine. In: Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retrieval (SIGIR), Seattle, Washington; 2006. p. 99-106.
76. Frakes W.B., Baeza-Yates R. Information retrieval: data structure and algorithms 1992, Prentice-Hall.
77. Salton G. A vector space model for information retrieval. CACM 1975, 18:613-620.
78. Muller H.M., Kenny E.E., Sternberg P.W. Textpresso: an ontology-based information retrieval and extraction system for biological literature. PLoS Biol 2004, 2:e309.
79. Deerwester S.C., Dumais S.T., Landauer T.K., Furnas G.W., Harshman R.A. Indexing by latent semantic analysis. J Am Soc Inform Sci 1990, 41(6):391-407.
80. Hofmann T. Probabilistic latent semantic indexing. In: Proceedings of the 22nd annual international ACM SIGIR conference on research and development in information retrieval, California, USA; 1999. p. 50-7.
81. Ponte J, Croft W. A language modeling approach to information retrieval. In: Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval, Melbourne, Australia; 1998. p. 275-81.
82. Sneiderman C.A., Demner-Fushman D., Fiszman M., Ide N.C., Rindflesch T.C. Knowledge-based methods to help clinicians find answers in MEDLINE. J Am Med Inform Assoc 2007, 14:772-780.
83. Srinivasan P., Rindflesch T. Exploring text mining from MEDLINE. Proc AMIA Symp 2002, 722-726.
84. Humphreys B.L., Lindberg D.A. The UMLS project: making the conceptual connection between users and the information they need. Bull Med Lib Assoc 1993, 81:170-177.
85. Ide N.C., Loane R.F., Demner-Fushman D. Essie: a concept-based search engine for structured biomedical text. J Am Med Inform Assoc 2007, 14:253-263.
86. Hovy E, Hermjakob U, Lin CY. The use of external knowledge in factoid QA. In: TREC 2001; 2001. p. 644-52.
87. Joachims T. Text categorization with support vector machines: learning with many relevant features. Lecture notes in computer science. Berlin/Heidelberg: Springer; 1997. p. 137-42.
88. Lafferty J, McCallum A, Pereira F. Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the eighteenth international conference on machine learning (ICML); 2001. p. 282-9.
89. Aronson A.R. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proc AMIA Symp 2001, 17-21.
90. Robertson S, Zaragoza H, Taylor M. Simple BM25 extension to multiple weighted fields. In: ACM CIKM; 2004.
91. Zaragoza H, Craswell N, Taylor M, Saria S, Robertson S. Microsoft Cambridge at TREC-13: web and HARD tracks. In: Proc of TREC 2004; 2004.
92. Tellex S, Katz B, Lin J. Quantitative evaluation of passage retrieval algorithms for question answering. In: Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval; 2003. p. 41-7.
93. Gobeill J, Ehrler F, Tbahriti I, Ruch P. Vocabulary-driven passage retrieval for question-answering in genomics. In: The sixteenth text retrieval conference, TREC, Gaithersburg, MD; 2007.
94. Liu X, Croft WB. Passage retrieval based on language models. In: Proc of CIKM; 2002. p. 375-82.
95. Hearst M.A. TextTiling: segmenting text into multi-paragraph subtopic passages. Comput Linguist 1997, 23:33-64.
96. Paterson M, Dancik V. Longest common subsequences. In: Proc of 19th MFCS, No. 841 in LNCS, vol. 841; 1994. p. 127-42.
97. Liu Feifan, Liu Yang. Exploring correlation between ROUGE and human evaluation on meeting summaries. Audio, speech, and language processing. IEEE Trans 2010;18:187-96.
98. Osinski S., Stefanowski J., Weiss D. Lingo: search results clustering algorithm based on singular value decomposition. Intell Inform Syst 2004, 359-368.
99. Hirschberg D.S. Algorithms for the longest common subsequence problem. J ACM 1977, 24:664-675.
100. Lin C. ROUGE: a package for automatic evaluation of summaries. In: Proceedings of the ACL workshop: text summarization braches out 2004; 2004. p. 74-81.
101. Ng RT, Zhou X. Scalable discovery of hidden emails from large folders. In: ACM SIGKDD'05; 2005. p. 544-9.
102. Zhang Y, Patrick J. Paraphrase identification by text canonicalization. In: Proceedings of the Australasian language technology workshop 2005; 2005.
103. Abdou S., Savoy J. Searching in Medline: query expansion and manual indexing evaluation. Inform Process Manage 2008, 44:781-789.
104. Zighelnic L, Kurland O. Query-drift prevention for robust query expansion. In: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval; 2008. p. 825-6.
105. Stokes N., Li Y., Cavedon L., Zobel J. Exploring criteria for successful query expansion in the genomic domain. Inform Retrieval 2009, 12:17-50.
106. Rinaldi F, Dowdall J, Schneider G, Persidis A. Answering questions in the genomics domain. ACL-2004 workshop on question answering in restricted domains, Barcelona, Spain; 2007.