Crowdsourcing Twitter annotations to identify first-hand experiences of prescription drug use

Journal of Biomedical Informatics

Volumn 58, Issue , 2015, Pages 280-287

  Alvaro, Nestor ^a,b   Conway, Mike ^c   Doan, Son ^d   Lofi, Christoph ^e   Overington, John ^f   Collier, Nigel ^g,h  

^a NATIONAL INSTITUTE OF INFORMATICS   (Japan)

^b GRADUATE UNIVERSITY FOR ADVANCED STUDIES   (Japan)

^c UNIVERSITY OF UTAH SCHOOL OF MEDICINE   (United States)

^d KAISER PERMANENTE   (United States)

^e TECHNICAL UNIVERSITY OF BRAUNSCHWEIG   (Germany)

^f Stratified Medical   (United Kingdom)

^g UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^h EUROPEAN BIOINFORMATICS INSTITUTE   (United Kingdom)

Author keywords

Crowdsourcing; Natural language processing; Pharmacovigilance; Twitter

Indexed keywords

ARTIFICIAL INTELLIGENCE; HOSPITAL DATA PROCESSING; LEARNING ALGORITHMS; LEARNING SYSTEMS; NATURAL LANGUAGE PROCESSING SYSTEMS; SUPERVISED LEARNING;

ADVERSE DRUG REACTION (ADRS); CROWDSOURCING; GENERALIZED LINEAR MODEL; MICRO-BLOGGING SERVICES; NATURAL LANGUAGE PROCESSING; PHARMACOVIGILANCE; SUPERVISED MACHINE LEARNING; TWITTER;

SOCIAL NETWORKING (ONLINE);

ADVERSE OUTCOME; ARTICLE; BAYES THEOREM; COMPUTER INTERFACE; COMPUTER LANGUAGE; COMPUTER MODEL; CROWDSOURCING; DRUG USE; HUMAN COMPUTER INTERACTION; MACHINE LEARNING; MATHEMATICAL MODEL; ONLINE SYSTEM; PRESCRIPTION; PRIORITY JOURNAL; SOCIAL MEDIA; HUMAN;

CROWDSOURCING; DRUG PRESCRIPTIONS; HUMANS; SOCIAL MEDIA;

EID: 84947918869     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2015.11.004     Document Type: Article

References (46)

1. Kohn L.T., Corrigan J.M., Donaldson M.S., et al. To Err is Human: Building a Safer Health System 2000, vol. 627. National Academies Press.
2. Lazarou J., Pomeranz B.H., Corey P.N. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA: J. Am. Med. Assoc. 1998, 279(15):1200-1205.
3. The International Conference on Harmonisation Expert Working Group, Guideline for Good Clinical Practice e6(r1), 1996. <>. http://private.ich.org/LOB/media/MEDIA482.pdf.
4. Office of the Commissioner, Medwatch: The FDA Safety Information and Adverse Event Reporting Program, 2014. <>. http://www.fda.gov/Safety/MedWatch/default.htm.
5. Hazell L., Shakir S.A. Under-reporting of adverse drug reactions. Drug Saf. 2006, 29(5):385-396.
6. Leaman R., Wojtulewicz L., Sullivan R., Skariah A., Yang J., Gonzalez G. Towards internet-age pharmacovigilance: extracting adverse drug reactions from user posts to health-related social networks. Proceedings of the 2010 Workshop on Biomedical Natural Language Processing 2010, 117-125. Association for Computational Linguistics.
7. Hanson C.L., Cannon B., Burton S., Giraud-Carrier C. An exploration of social circles and prescription drug abuse through Twitter. J. Med. Internet Res. 2013, 15(9).
8. FDA, Guidance for Industry Internet/Social Media Platforms with Character Space Limitations Presenting Risk and Benefit Information for Prescription Drugs and Medical Devices, June 2014. <>. http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm401087.pdf.
9. Medicines and Healthcare Products Regulatory Agency (MHRA). Press Release: UK Regulator Leads Innovative EU Project on the use of Smartphones and Social Media for Drug Safety Information, November 2015. <>. http://www.imi.europa.eu/content/web-radr.
10. E.M. Agency, European Medicines Agency. Guideline on Good Pharmacovigilance Practices (GVP), 2013. <>. http://www.ema.europa.eu/ema/index.jsp?curl=pages/regulation/document_listing/document_listing_000345.jsp.
11. E.M. Agency, Guideline on Good Pharmacovigilance Practices (GVP) Module VI Management and Reporting of Adverse Reactions to Medicinal Products (rev 1), 2014. <>. http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2014/09/WC500172402.pdf.
12. United States Securities and Exchange Commission, Form s-1. Registration statement. Twitter, inc., October 2013. <>. http://www.sec.gov/Archives/edgar/data/1418091/000119312513390321/d564001ds1.htm.
13. Nikfarjam A., Gonzalez G.H. Pattern mining for extraction of mentions of adverse drug reactions from user comments. AMIA Annual Symposium Proceedings 2011, vol. 2011:1019. American Medical Informatics Association.
14. González-Rubio F., Calderón-Larrañaga A., Poblador-Plou B., Navarro-Pemán C., López-Cabañas A., Prados-Torres A. Underreporting of recognized adverse drug reactions by primary care physicians: an exploratory study. Pharmacoepidem. Drug Saf. 2011, 20(12):1287-1294.
15. Revealed: The Demographic Trends for Every Social Network, October 2014. <>. http://www.businessinsider.com/2014-social-media-demographics-update-2014-9.
16. Schneeweiss S., Patrick A.R., Solomon D.H., Dormuth C.R., Miller M., Mehta J., Lee J.C., Wang P.S. Comparative safety of antidepressant agents for children and adolescents regarding suicidal acts. Pediatrics 2010, peds-2009.
17. Hanson C.L., Burton S.H., Giraud-Carrier C., West J.H., Barnes M.D., Hansen B. Tweaking and tweeting: exploring Twitter for nonmedical use of a psychostimulant drug (Adderall) among college students. J. Med. Internet Res. 2013, 15(4).
18. Freifeld C.C., Brownstein J.S., Menone C.M., Bao W., Filice R., Kass-Hout T., Dasgupta N. Digital drug safety surveillance: monitoring pharmaceutical products in Twitter. Drug Saf. 2014, 37(5):343-350.
19. U. Food, D. Administration, FDA Adverse Event Reporting System (FAERS), 2015. <>. http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Surveillance/AdverseDrugEffects/.
20. Ragan C.I., Bard I., Singh I. What should we do about student use of cognitive enhancers? An analysis of current evidence. Neuropharmacology 2013, 64:588-595.
21. White N.C., Litovitz T., Clancy C. Suicidal antidepressant overdoses: a comparative analysis by antidepressant type. J. Med. Toxicol. 2008, 4(4):238-250.
22. Twitter Inc., Get Statuses/Sample, September 2014. <>. http://https://dev.twitter.com/streaming/reference/get/statuses/sample.
23. D. Sell, FDA Warns of Counterfeit Adderall, October 2012. <>. http://articles.philly.com/2012-05-31/business/31900817_1_rogue-websites-and-distributors-generic-versions-adderall.
24. D. O'Neil, John Moffitt on Adderall: 'it was a total mistake', November 2012. <>. http://seattletimes.com/html/seahawksblog/2019783660_adderall28.html.
25. Aurobindo Pharma Gets USFDA nod for Modafinil Tablets, September 2012. <>. http://articles.economictimes.indiatimes.com/2012-09-28/news/34148290_1_aurolife-pharma-llc-usfda-nod-aurobindo-pharma.
26. Cohen J. A coefficient of agreement for nominal scales. Educ. Psychol. Measur. 1960, 20(1):37-46.
27. Fleiss J.L., Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ. Psychol. Meas. 1973.
28. M. Gamer, Package 'irr', 2012. <>. http://cran.r-project.org/web/packages/irr/irr.pdf.
29. Myslín M., Zhu S.-H., Chapman W., Conway M. Using Twitter to examine smoking behavior and perceptions of emerging tobacco products. J. Med. Internet Res. 2013, 15(8).
30. Conway M., Doan S., Kawazoe A., Collier N. Classifying disease outbreak reports using n-grams and semantic features. Int. J. Med. Inform. 2009, 78(12):e47-e58.
31. Y. Tewari, R. Kawad, Real-Time Topic Modeling of Microblogs, March 2013. <>. http://www.oracle.com/technetwork/articles/java/micro-1925135.html.
32. R. Mehrotra, S. Sanner, W. Buntine, L. Xie, Improving LDA Topic Models for Microblogs via Tweet Pooling and Automatic Labeling, 2013.
33. Blei D.M., Ng A.Y., Jordan M.I. Latent dirichlet allocation. J. Mach. Learn. Res. 2003, 3:993-1022.
34. Lau J.H., Grieser K., Newman D., Baldwin T. Automatic labelling of topic models. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 1 2011, 1536-1545. Association for Computational Linguistics.
35. Y. Yang, J.O. Pedersen, A comparative study on feature selection in text categorization, in: ICML, vol. 97, 1997, pp. 412-420.
36. Hong L., Davison B.D. Empirical study of topic modeling in Twitter. Proceedings of the First Workshop on Social Media Analytics 2010, 80-88. ACM.
37. P. Romanski, Package "FSelector, February 2013. <>. http://cran.r-project.org/web/packages/FSelector/FSelector.pdf.
38. Spearman C. The proof and measurement of association between two things. Am. J. Psychol. 1904, 15(1):72-101.
39. Kendall M.G. A new measure of rank correlation. Biometrika 1938, 81-93.
40. C. Dancy, J. Reidy, Statistics without maths for psychology, IEEE Statistics without maths for psychology, 2004.
41. Nowak S., Rüger S. How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation. Proceedings of the International Conference on Multimedia Information Retrieval 2010, 557-566. ACM.
42. Brown L.D., Cai T.T., DasGupta A. Interval estimation for a binomial proportion. Stat. Sci. 2001, 101-117.
43. How to Calculate a Confidence Score, July 2014. <>. http://success.crowdflower.com/customer/portal/articles/1295977-how-to-calculate-a-confidence-score.
44. CrowdFlower, Get Results - How to Calculate a Confidence Score, 2015. <>. http://https://success.crowdflower.com/hc/en-us/articles/201855939-Get-Results-How-to-Calculate-a-Confidence-Score.
45. M. Kuhn, Package 'caret', August 2014. <>. http://cran.r-project.org/web/packages/caret/caret.pdf.
46. D.M. Powers, Evaluation: from Precision, Recall and F-measure to ROC, Informedness, Markedness and Correlation, 2011.