Automatically detecting failures in natural language processing tools for online community text

Journal of Medical Internet Research

Volumn 17, Issue 8, 2015, Pages

  Park, Albert ^a   Hartzler, Andrea L ^b   Huh, Jina ^c   McDonald, David W ^d   Pratt, Wanda ^d  

^a UNIVERSITY OF WASHINGTON SCHOOL OF MEDICINE   (United States)

^b GROUP HEALTH RESEARCH INSTITUTE   (United States)

^c UNIVERSITY OF CALIFORNIA   (United States)

^d UNIVERSITY OF WASHINGTON   (United States)

Author keywords

Automatic data processing; Information extraction; Natural language processing; Quantitative evaluation; UMLS

Indexed keywords

HUMAN; INFORMATION PROCESSING; INTERNET; NATURAL LANGUAGE PROCESSING;

AUTOMATIC DATA PROCESSING; HUMANS; INTERNET; NATURAL LANGUAGE PROCESSING;

EID: 84940645530     PISSN: None     EISSN: 14388871     Source Type: Journal    
DOI: 10.2196/jmir.4612     Document Type: Article

References (52)

1. Fox S, Rainie L. Pew Research Center Internet, Science & Tech. 2014 Feb 27. The Web at 25 in the US The overall verdict: The internet has been a plus for society and an especially good thing for individual users URL: http://www.pewinternet.org/2014/02/27/the-web-at-25-in-the-u-s/ [accessed 2015-04-23] [WebCite Cache ID 6Y5L8D2oE]
2. Fox S. Pew Research Center Internet, Science & Tech. 2005 May 17. Health Information online: Eight in ten internet users have looked for health information online, with increased interest in diet, fitness, drugs, health insurance, experimental treatments, and particular doctors and hospitals URL: http://www.pewinternet.org/2005/05/17/health-information-online/ [accessed 2015-04-23] [WebCite Cache ID 6Y5JQkmrG]
3. Fox S. Pew Research Center Internet, Science & Tech. 2011 Feb 28. Peer-to-peer healthcare: The internet gives patients and caregivers access not only to information, but also to each other URL: http://www.pewinternet.org/2011/02/28/peer-to-peer-health-care-2/ [accessed 2015-04-25] [WebCite Cache ID 6Y5JjmduI]
4. Eysenbach G. Medicine 2.0: social networking, collaboration, participation, apomediation, and openness. J Med Internet Res 2008;10(3):e22 [FREE Full text] [doi: 10.2196/jmir.1030] [Medline: 18725354]
5. Starbird K, Palen L. 'Voluntweeters': Self-Organizing by Digital Volunteers in Times of Crisis. : ACM; 2011 Presented at: ACM CHI Conference on Human Factors in Computing Systems; May 07-12, 2011; Vancouver, BC p. 1071-1080 URL: http://dl.acm.org/citation.cfm?id=1979102 [WebCite Cache] [doi: 10.1145/1978942.1979102]
6. Wicks P, Vaughan TE, Massagli MP, Heywood J. Accelerated clinical discovery using self-reported patient data collected online and a patient-matching algorithm. Nat Biotechnol 2011 May;29(5):411-414. [doi: 10.1038/nbt.1837] [Medline: 21516084]
7. Wicks P, Massagli M, Frost J, Brownstein C, Okun S, Vaughan T, et al. Sharing health data for better outcomes on PatientsLikeMe. J Med Internet Res 2010;12(2):e19 [FREE Full text] [doi: 10.2196/jmir.1549] [Medline: 20542858]
8. Frost JH, Massagli MP. Social uses of personal health information within PatientsLikeMe, an online patient community: what can happen when patients have access to one another's data. J Med Internet Res 2008;10(3):e15 [FREE Full text] [doi: 10.2196/jmir.1053] [Medline: 18504244]
9. Wen M, Rose C. Understanding participant behavior trajectories in online health support groups using automatic extraction methods. : ACM; 2012 Presented at: 17th ACM international conference on Supporting group work; October 27-31, 2012; Sanibel Island, Florida p. 179-188 URL: http://dl.acm.org/citation.cfm?id=2389205 [WebCite Cache] [doi: 10.1145/2389176.2389205]
10. Wang Y, Kraut R, Levine JM. To Stay or Leave ? The Relationship of Emotional and Informational Support to Commitment in Online Health Support Groups. : ACM; 2012 Presented at: ACM conference on Computer Supported Cooperative Work (CSCW); Feb 11-15, 2012; Seattle, Washington. [doi: 10.1145/2145204.2145329]
11. Chee BW, Berlin R, Schatz B. Predicting adverse drug events from personal health messages. AMIA Annu Symp Proc 2011;2011:217-226 [FREE Full text] [Medline: 22195073]
12. MacLean DL, Heer J. Identifying medical terms in patient-authored text: a crowdsourcing-based approach. J Am Med Inform Assoc 2013;20(6):1120-1127 [FREE Full text] [doi: 10.1136/amiajnl-2012-001110] [Medline: 23645553]
13. Maclean DL, Gupta S, Lembke A, Manning C, Heer J. Forum77: An Analysis of an Online Health Forum Dedicated to Addiction Recovery. : ACM; 2015 Presented at: 18th ACM conference on Computer supported cooperative work & social computing (CSCW); Mar 14-18, 2015; Vancouver, BC. [doi: 10.1145/2675133.2675146]
14. De Choudhury M, Gamon M, Counts S, Horvitz E. Predicting depression via social media. 2013 Presented at: 7th International AAAI Conference on Weblogs and SocialMedia (ICWSM); Jul 8-10, 2013; Boston, Massachusetts URL: http://www.aaai.org/ocs/index.php/ICWSM/ICWSM13/paper/viewFile/6124/6351 [WebCite Cache]
15. Huh J, Yetisgen-Yildiz M, Pratt W. Text classification for assisting moderators in online health communities. J Biomed Inform 2013 Dec;46(6):998-1005 [FREE Full text] [doi: 10.1016/j.jbi.2013.08.011] [Medline: 24025513]
16. Savova GK, Masanz JJ, Ogren PV, Zheng J, Sohn S, Kipper-Schuler KC, et al. Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. J Am Med Inform Assoc 2010;17(5):507-513 [FREE Full text] [doi: 10.1136/jamia.2009.001560] [Medline: 20819853]
17. Musen MA, Noy NF, Shah NH, Whetzel PL, Chute CG, Story M, et al. The National Center for Biomedical Ontology. J Am Med Inform Assoc 2012;19(2):190-195 [FREE Full text] [doi: 10.1136/amiajnl-2011-000523] [Medline: 22081220]
18. Jonquet C, Shah NH, Musen MA. The open biomedical annotator. Summit on Translat Bioinforma 2009;2009:56-60. [Medline: 21347171]
19. Friedman C, Hripcsak G, DuMouchel W, Johnson SB, Clayton PD. Natural language processing in an operational clinical information system. Nat Lang Eng 2008 Sep 12;1(01):-. [doi: 10.1017/S1351324900000061]
20. Friedman C. Towards a comprehensive medical language processing system: methods and issues. Proc AMIA Annu Fall Symp 1997:595-599. [Medline: 9357695]
21. Chen L, Friedman C. Extracting phenotypic information from the literature via natural language processing. Stud Health Technol Inform 2004;107(Pt 2):758-762. [Medline: 15360914]
22. Aronson AR, Lang F. An overview of MetaMap: historical perspective and recent advances. J Am Med Inform Assoc 2010;17(3):229-236 [FREE Full text] [doi: 10.1136/jamia.2009.002733] [Medline: 20442139]
23. Humphreys BL, Lindberg DA, Schoolman HM, Barnett GO. The Unified Medical Language System: an informatics research collaboration. J Am Med Inform Assoc 1998;5(1):1-11 [FREE Full text] [Medline: 9452981]
24. Chen Y, Perl Y, Geller J, Cimino JJ. Analysis of a study of the users, uses, and future agenda of the UMLS. J Am Med Inform Assoc 2007;14(2):221-231 [FREE Full text] [doi: 10.1197/jamia.M2202] [Medline: 17213497]
25. Zeng Q, Kogan S, Ash N, Greenes RA. Patient and clinician vocabulary: how different are they? Stud Health Technol Inform 2001;84(Pt 1):399-403. [Medline: 11604772]
26. Smith CA, Wicks PJ. PatientsLikeMe: Consumer health vocabulary as a folksonomy. AMIA Annu Symp Proc 2008:682-686 [FREE Full text] [Medline: 18999004]
27. Keselman A, Smith CA, Divita G, Kim H, Browne AC, Leroy G, et al. Consumer health concepts that do not map to the UMLS: where do they fit? J Am Med Inform Assoc 2008;15(4):496-505 [FREE Full text] [doi: 10.1197/jamia.M2599] [Medline: 18436906]
28. Zeng QT, Crowell J, Plovnick RM, Kim E, Ngo L, Dibble E. Assisting consumer health information retrieval with query recommendations. J Am Med Inform Assoc 2006;13(1):80-90. [doi: 10.1197/jamia.M1820] [Medline: 16221944]
29. Brennan PF, Aronson AR. Towards linking patients and clinical information: detecting UMLS concepts in e-mail. J Biomed Inform 2003;36(4-5):334-341 [FREE Full text] [Medline: 14643729]
30. Smith CA, Stavri PZ, Chapman WW. In their own words? A terminological analysis of e-mail to a cancer information service. Proc AMIA Symp 2002:697-701 [FREE Full text] [Medline: 12463914]
31. Zeng QT, Tse T. Exploring and developing consumer health vocabularies. J Am Med Inform Assoc 2006;13(1):24-29 [FREE Full text] [doi: 10.1197/jamia.M1761] [Medline: 16221948]
32. Elhadad N, Zhang S, Driscoll P, Brody S. Characterizing the Sublanguage of Online Breast Cancer Forums for Medications, Symptoms, and Emotions. 2014 Presented at: AMIA Annual Fall Symposium; Nov. 15-19, 2014; Washington DC.
33. Strauss A, Corbin J. Basics of qualitative research. In: Basics of qualitative research: grounded theory procedures and techniques. Newbury Park, CA: Sage Publications; 1990.
34. Divita G, Tse T, Roth L. Failure analysis of MetaMap Transfer (MMTx). Stud Health Technol Inform 2004;107(Pt 2):763-767. [Medline: 15360915]
35. Pratt W, Yetisgen-Yildiz M. A study of biomedical concept identification: MetaMap vs. people. AMIA Annu Symp Proc 2003:529-533 [FREE Full text] [Medline: 14728229]
36. Kang N, Singh B, Afzal Z, van Mulligen EM, Kors JA. Using rule-based natural language processing to improve disease normalization in biomedical text. J Am Med Inform Assoc 2013;20(5):876 881 [FREE Full text] [doi: 10.1136/amiajnl-2012-001173] [Medline: 23043124]
37. Danescu-Niculescu-Mizil C, West R, Jurafsky D, Leskovec J, Potts C. No Country for Old Members: User Lifecycle and Linguistic Change in Online Communities. 2013 Presented at: 22nd Int Conf World Wide Web; May 13-17, 2013; Rio de Janeiro, Brazil p. 307-317.
38. McCray AT, Ide NC, Loane RR, Tse T. Strategies for supporting consumer health information seeking. Stud Health Technol Inform 2004;107(Pt 2):1152-1156. [Medline: 15360993]
39. McCray AT, Ide NC. Design and implementation of a national clinical trials registry. J Am Med Inform Assoc 2000;7(3):313-323 [FREE Full text] [Medline: 10833169]
40. Smith L, Rindflesch T, Wilbur WJ. MedPost: a part-of-speech tagger for bioMedical text. Bioinformatics 2004 Sep 22;20(14):2320-2321 [FREE Full text] [doi: 10.1093/bioinformatics/bth227] [Medline: 15073016]
41. Jansen E, James V. NetLingo: the Internet dictionary. 2002. URL: http://www.netlingo.com/ [accessed 2015-04-20] [WebCite Cache ID 6XwJyMzTG]
42. Bureau U census. Frequently Occurring Names in the US. 1990. URL: http://deron.meranda.us/data/census-derived-all-first.txt [accessed 2015-04-20] [WebCite Cache ID 6XwJ9Enlw]
43. Bamford S, Dawson E, Forbes S, Clements J, Pettett R, Dogan A, et al. The COSMIC (Catalogue of Somatic Mutations in Cancer) database and website. Br J Cancer 2004 Jul 19;91(2):355-358 [FREE Full text] [doi: 10.1038/sj.bjc.6601894] [Medline: 15188009]
44. Toutanova K, Klein D, Manning C, Singer Y. Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network. 2003 Presented at: 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1; May 2003; Edmonton, AB p. 173-180 URL: http://dl.acm.org/citation.cfm?id=1073478 [WebCite Cache] [doi: 10.3115/1073445.1073478]
45. De Marneffe MC, MacCartney B, Manning CD. Generating Typed Dependency Parses from Phrase Structure Parses. 2006 Presented at: LREC; May 22-28, 2006; Genoa, Italy.
46. Schwartz AS, Hearst MA. A simple algorithm for identifying abbreviation definitions in biomedical text. Pac Symp Biocomput 2003:451-462 [FREE Full text] [Medline: 12603049]
47. Chapman WW, Bridewell W, Hanbury P, Cooper GF, Buchanan BG. A simple algorithm for identifying negated findings and diseases in discharge summaries. J Biomed Inform 2001 Oct;34(5):301-310 [FREE Full text] [doi: 10.1006/jbin.2001.1029] [Medline: 12123149]
48. Baumgartner WA, Lu Z, Johnson HL, Caporaso JG, Paquette J, Lindemann A, et al. Concept recognition for extracting protein interaction relations from biomedical text. Genome Biol 2008;9 Suppl 2:S9 [FREE Full text] [doi: 10.1186/gb-2008-9-s2-s9] [Medline: 18834500]
49. Jimeno-Yepes A, Berlanga-Llavori R, Rebholz-Schuhmann D. Ontology refinement for improved information retrieval. Information Processing & Management 2010 Jul;46(4):426-435. [doi: 10.1016/j.ipm.2009.05.008]
50. Query Suggestion Service. URL: http://www.google.com/support/enterprise/static/gsa/docs/admin/70/gsa-doc-set/xml-reference/query-suggestion.html [accessed 2014-10-13] [WebCite Cache ID 6TIeuLgUB]
51. MetaMap Updates. URL: http://metamap.nlm.nih.gov/MainDownload.shtml [accessed 2014-10-05] [WebCite Cache ID 6T6e318kp]
52. Dredze M. How Social Media Will Change Public Health. IEEE Intell. Syst 2012 Jul;27(4):81-84 [FREE Full text] [doi: 10.1109/MIS.2012.76]