Automatically appraising the credibility of vaccine-related web pages shared on social media: A twitter surveillance study

Journal of Medical Internet Research

Volumn 21, Issue 11, 2019, Pages

  Shah, Zubair ^a,b   Surian, Didi ^a   Dyda, Amalie ^a   Coiera, Enrico ^a   Mandl, Kenneth D ^c,d   Dunn, Adam G ^a,d  

^a MACQUARIE UNIVERSITY   (Australia)

^b HAMAD BIN KHALIFA UNIVERSITY   (Qatar)

^c HARVARD MEDICAL SCHOOL   (United States)

^d BOSTON CHILDREN S HOSPITAL   (United States)

Author keywords

Credibility appraisal; Health misinformation; Machine learning; Social media

Indexed keywords

VACCINE;

EPIDEMIOLOGICAL MONITORING; HUMAN; MACHINE LEARNING; RETROSPECTIVE STUDY; SOCIAL MEDIA; SOCIAL NETWORK;

EPIDEMIOLOGICAL MONITORING; HUMANS; MACHINE LEARNING; RETROSPECTIVE STUDIES; SOCIAL MEDIA; SOCIAL NETWORKING; VACCINES;

EID: 85074544598     PISSN: None     EISSN: 14388871     Source Type: Journal    
DOI: 10.2196/14007     Document Type: Article

References (58)

1. Lazer DM, Baum MA, Benkler Y, Berinsky AJ, Greenhill KM, Menczer F, et al. The science of fake news. Science 2018 Mar 9;359(6380):1094-1096. [doi: 10.1126/science.aao2998] [Medline: 29590025]
2. Budak C, Agrawal D, El AA. Limiting the Spread of Misinformation in Social Networks. In: Proceedings of the 20th International Conference on World Wide Web. 2011 Presented at: WWW'11; March 28-April 1, 2011; Hyderabad, India p. 665-674. [doi: 10.1145/1963405.1963499]
3. Mocanu D, Rossi L, Zhang Q, Karsai M, Quattrociocchi W. Collective attention in the age of (mis)information. Comput Hum Behav 2015 Oct;51:1198-1204. [doi: 10.1016/j.chb.2015.01.024]
4. Tambuscio M, Ruffo G, Flammini A, Menczer F. Fact-Checking Effect on Viral Hoaxes: A Model of Misinformation Spread in Social Networks. In: Proceedings of the 24th International Conference on World Wide Web. 2015 Presented at: WWW'15; May 18-22, 2015; Florence, Italy p. 977-982. [doi: 10.1145/2740908.2742572]
5. Kumar S, West R, Leskovec J. Disinformation on the Web: Impact, Characteristics, and Detection of Wikipedia Hoaxes. In: Proceedings of the 25th International Conference on World Wide Web. 2016 Presented at: WWW '16; April 11-15, 2016; Montreal, Canada p. 591-602. [doi: 10.1145/2872427.2883085]
6. Vosoughi S, Roy D, Aral S. The spread of true and false news online. Science 2018 Mar 9;359(6380):1146-1151. [doi: 10.1126/science.aap9559] [Medline: 29590045]
7. Grinberg N, Joseph K, Friedland L, Swire-Thompson B, Lazer D. Fake news on Twitter during the 2016 US presidential election. Science 2019 Jan 25;363(6425):374-378. [doi: 10.1126/science.aau2706] [Medline: 30679368]
8. del Vicario M, Bessi A, Zollo F, Petroni F, Scala A, Caldarelli G, et al. The spreading of misinformation online. Proc Natl Acad Sci U S A 2016 Jan 19;113(3):554-559 [FREE Full text] [doi: 10.1073/pnas.1517441113] [Medline: 26729863]
9. Weaver JB, Thompson NJ, Weaver SS, Hopkins GL. Healthcare non-adherence decisions and internet health information. Comput Hum Behav 2009 Nov;25(6):1373-1380. [doi: 10.1016/j.chb.2009.05.011]
10. Lewandowsky S, Ecker UK, Seifert CM, Schwarz N, Cook J. Misinformation and its correction: continued influence and successful debiasing. Psychol Sci Public Interest 2012 Dec;13(3):106-131. [doi: 10.1177/1529100612451018] [Medline: 26173286]
11. Schwitzer G. A review of features in internet consumer health decision-support tools. J Med Internet Res 2002;4(2):E11 [FREE Full text] [doi: 10.2196/jmir.4.2.e11] [Medline: 12554558]
12. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. Br Med J 2002 Mar 9;324(7337):573-577 [FREE Full text] [doi: 10.1136/bmj.324.7337.573] [Medline: 11884321]
13. Reuters Institute Digital News Report. 2019. Reuters Institute for the Study of Journalism, University of OxfordURL: http://www.digitalnewsreport.org/ [accessed 2019-06-29]
14. Winters M, Larsson A, Kowalski J, Sundberg CJ. The association between quality measures of medical university press releases and their corresponding news stories-important information missing. PLoS One 2019;14(6):e0217295 [FREE Full text] [doi: 10.1371/journal.pone.0217295] [Medline: 31188838]
15. Yavchitz A, Boutron I, Bafeta A, Marroun I, Charles P, Mantz J, et al. Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study. PLoS Med 2012;9(9):e1001308 [FREE Full text] [doi: 10.1371/journal.pmed.1001308] [Medline: 22984354]
16. Haneef R, Ravaud P, Baron G, Ghosn L, Boutron I. Factors associated with online media attention to research: a cohort study of articles evaluating cancer treatments. Res Integr Peer Rev 2017;2:9 [FREE Full text] [doi: 10.1186/s41073-017-0033-z] [Medline: 29451556]
17. Sumner P, Vivian-Griffiths S, Boivin J, Williams A, Venetis CA, Davies A, et al. The association between exaggeration in health related science news and academic press releases: retrospective observational study. Br Med J 2014 Dec 9;349:g7015 [FREE Full text] [doi: 10.1136/bmj.g7015] [Medline: 25498121]
18. Grundy Q, Dunn AG, Bourgeois FT, Coiera E, Bero L. Prevalence of disclosed conflicts of interest in biomedical research and associations with journal impact factors and altmetric scores. J Am Med Assoc 2018 Jan 23;319(4):408-409 [FREE Full text] [doi: 10.1001/jama.2017.20738] [Medline: 29362787]
19. Bernstam EV, Shelton DM, Walji M, Meric-Bernstam F. Instruments to assess the quality of health information on the World Wide Web: what can our patients actually use? Int J Med Inform 2005 Jan;74(1):13-19. [doi: 10.1016/j.ijmedinf.2004.10.001] [Medline: 15626632]
20. Steffens M, Dunn A, Leask J. Meeting the challenges of reporting on public health in the new media landscape. Aust J Rev 2017;39(2):119-132 [FREE Full text]
21. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999 Feb;53(2):105-111 [FREE Full text] [doi: 10.1136/jech.53.2.105] [Medline: 10396471]
22. Zeraatkar D, Obeda M, Ginsberg JS, Hirsh J. The development and validation of an instrument to measure the quality of health research reports in the lay media. BMC Public Health 2017 Apr 20;17(1):343 [FREE Full text] [doi: 10.1186/s12889-017-4259-y] [Medline: 28427426]
23. HealthNewsReview. Our Review Criteria URL: https://www.healthnewsreview.org/about-us/review-criteria/ [accessed 2019-06-29]
24. Larson H. The biggest pandemic risk? Viral misinformation. Nature 2018 Oct;562(7727):309. [doi: 10.1038/d41586-018-07034-4] [Medline: 30327527]
25. Qazvinian V, Rosengren E, Radev D, Mei Q. Rumor Has It: Identifying Misinformation in Microblogs. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2011 Presented at: EMNL '11; July 27-31, 2011; Edinburgh, United Kingdom p. 1589-1599.
26. Zhao Z, Resnick P, Mei Q. Enquiring Minds: Early Detection of Rumors in Social Media from Enquiry Posts. In: Proceedings of the 24th International Conference on World Wide Web. 2015 Presented at: WWW'15; May 18-22, 2015; Florence, Italy p. 1395-1405. [doi: 10.1145/2736277.2741637]
27. Vosoughi S, Mohsenvand M, Roy D. Rumor gauge: predicting the veracity of rumors on Twitter. ACM Trans Knowl Discov Data 2017 Aug 21;11(4):1-36. [doi: 10.1145/3070644]
28. Ma J, Gao W, Mitra P, Kwon S, Jansen BJ, Wong KF, et al. Detecting Rumors From Microblogs With Recurrent Neural Networks. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. 2016 Presented at: IJCAI'16; July 09-15, 2016; New York, United States p. 3818-3824.
29. Ruchansky N, Seo S, Liu Y. CSI: A Hybrid Deep Model for Fake News Detection. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 2017 Presented at: CIKM'17; November 06-10, 2017; Singapore, Singapore p. 797-806. [doi: 10.1145/3132847.3132877]
30. Broniatowski DA, Jamison AM, Qi S, AlKulaib L, Chen T, Benton A, et al. Weaponized health communication: Twitter bots and Russian trolls amplify the vaccine debate. Am J Public Health 2018 Oct;108(10):1378-1384. [doi: 10.2105/AJPH.2018.304567] [Medline: 30138075]
31. Dredze M, Broniatowski DA, Smith MC, Hilyard KM. Understanding vaccine refusal: why we need social media now. Am J Prev Med 2016 Apr;50(4):550-552 [FREE Full text] [doi: 10.1016/j.amepre.2015.10.002] [Medline: 26655067]
32. Salathé M, Khandelwal S. Assessing vaccination sentiments with online social media: implications for infectious disease dynamics and control. PLoS Comput Biol 2011 Oct;7(10):e1002199 [FREE Full text] [doi: 10.1371/journal.pcbi.1002199] [Medline: 22022249]
33. Du J, Tang L, Xiang Y, Zhi D, Xu J, Song H, et al. Public perception analysis of tweets during the 2015 measles outbreak: comparative study using convolutional neural network models. J Med Internet Res 2018 Jul 9;20(7):e236 [FREE Full text] [doi: 10.2196/jmir.9413] [Medline: 29986843]
34. Chew C, Eysenbach G. Pandemics in the age of Twitter: content analysis of tweets during the 2009 H1N1 outbreak. PLoS One 2010 Nov 29;5(11):e14118 [FREE Full text] [doi: 10.1371/journal.pone.0014118] [Medline: 21124761]
35. Dunn AG, Leask J, Zhou X, Mandl KD, Coiera E. Associations between exposure to and expression of negative opinions about human papillomavirus vaccines on social media: an observational study. J Med Internet Res 2015 Jun 10;17(6):e144 [FREE Full text] [doi: 10.2196/jmir.4343] [Medline: 26063290]
36. Dunn AG, Surian D, Leask J, Dey A, Mandl KD, Coiera E. Mapping information exposure on social media to explain differences in HPV vaccine coverage in the United States. Vaccine 2017 May 25;35(23):3033-3040 [FREE Full text] [doi: 10.1016/j.vaccine.2017.04.060] [Medline: 28461067]
37. Surian D, Nguyen DQ, Kennedy G, Johnson M, Coiera E, Dunn AG. Characterizing Twitter discussions about HPV vaccines using topic modeling and community detection. J Med Internet Res 2016 Aug 29;18(8):e232 [FREE Full text] [doi: 10.2196/jmir.6045] [Medline: 27573910]
38. Du J, Cunningham RM, Xiang Y, Li F, Jia Y, Boom JA, et al. Leveraging deep learning to understand health beliefs about the Human Papillomavirus Vaccine from social media. NPJ Digit Med 2019;2:27 [FREE Full text] [doi: 10.1038/s41746-019-0102-4] [Medline: 31304374]
39. Shao C, Ciampaglia G, Flammini A, Menczer F. Hoaxy: A Platform for Tracking Online Misinformation. In: Proceedings of the 25th International Conference Companion on World Wide Web. 2016 Presented at: WWW'16; April 11-15, 2016; Montréal, Canada p. 745-750. [doi: 10.1145/2872518.2890098]
40. Dunn AG, Mandl KD, Coiera E. Social media interventions for precision public health: promises and risks. NPJ Digit Med 2018;1:47 [FREE Full text] [doi: 10.1038/s41746-018-0054-0] [Medline: 30854472]
41. Colditz JB, Chu K, Emery SL, Larkin CR, James AE, Welling J, et al. Toward real-time infoveillance of Twitter health messages. Am J Public Health 2018 Aug;108(8):1009-1014. [doi: 10.2105/AJPH.2018.304497] [Medline: 29927648]
42. Coiera E. Information epidemics, economics, and immunity on the internet. We still know so little about the effect of information on public health. Br Med J 1998 Nov 28;317(7171):1469-1470 [FREE Full text] [doi: 10.1136/bmj.317.7171.1469] [Medline: 9831570]
43. Google Code. 2018. Language Detection URL: https://code.google.com/p/language-detection/ [accessed 2019-06-29]
44. Centers for Disease Control and Prevention. 2009. Simply Put; A Guide for Creating Easy-to-understand Materials. Third EditionURL: https://stacks.cdc.gov/view/cdc/11938 [accessed 2019-06-29]
45. Moynihan R, Bero L, Ross-Degnan D, Henry D, Lee K, Watkins J, et al. Coverage by the news media of the benefits and risks of medications. N Engl J Med 2000 Jun 1;342(22):1645-1650. [doi: 10.1056/NEJM200006013422206] [Medline: 10833211]
46. Association of Health Care Journalists. Statement of Principles of the Association of Health Care Journalists URL: https://healthjournalism.org/secondarypage-details.php?id=56 [accessed 2019-06-29]
47. Wiggers J. The University of Newcastle. Media Doctor URL: https://www.newcastle.edu.au/research-and-innovation/centre/chb/research/media-doctor [accessed 2019-06-29]
48. Betsch C, Rossmann C, Habersaat KB, Pfeiffer D, Holtmann C, Korn L. World Health Organization Europe. 2017. Vaccination and Trust (2017) URL: http://www.euro.who.int/en/health-topics/disease-prevention/vaccines-and-immunization/publications/2017/vaccination-and-trust-2017 [accessed 2019-06-29]
49. Merity S, Keskar NS, Socher R. arXiv. 2017. Regularizing and optimizing LSTM language models URL: https://arxiv.org/abs/1708.02182 [accessed 2019-09-19]
50. Howard J, Ruder S. Universal Language Model Fine-Tuning for Text Classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018 Presented at: ACL'18; 2018; Melbourne, Australia p. 328-339. [doi: 10.18653/v1/P18-1031]
51. Mitra T, Wright G, Gilbert E. A Parsimonious Language Model of Social Media Credibility Across Disparate Events. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 2017 Presented at: CSCW'17; February 25-March 1, 2017; Portland, United States p. 126-145. [doi: 10.1145/2998181.2998351]
52. Tomeny TS, Vargo CJ, El-Toukhy S. Geographic and demographic correlates of autism-related anti-vaccine beliefs on Twitter, 2009-15. Soc Sci Med 2017 Oct;191:168-175 [FREE Full text] [doi: 10.1016/j.socscimed.2017.08.041] [Medline: 28926775]
53. Vraga EK, Bode L. Using expert sources to correct health misinformation in social media. J Sci Commun 2017 Sep 14;39(5):621-645. [doi: 10.1177/1075547017731776]
54. Bode L, Vraga EK. See something, say something: correction of global health misinformation on social media. Health Commun 2018 Sep;33(9):1131-1140. [doi: 10.1080/10410236.2017.1331312] [Medline: 28622038]
55. Pennycook G, Cannon TD, Rand DG. Prior exposure increases perceived accuracy of fake news. J Exp Psychol Gen 2018 Dec;147(12):1865-1880. [doi: 10.1037/xge0000465] [Medline: 30247057]
56. Briones R, Nan X, Madden K, Waks L. When vaccines go viral: an analysis of HPV vaccine coverage on YouTube. Health Commun 2012;27(5):478-485. [doi: 10.1080/10410236.2011.610258] [Medline: 22029723]
57. Madathil KC, Rivera-Rodriguez AJ, Greenstein JS, Gramopadhye AK. Healthcare information on YouTube: a systematic review. Health Informatics J 2015 Sep;21(3):173-194. [doi: 10.1177/1460458213512220] [Medline: 24670899]
58. Venkatraman A, Garg N, Kumar N. Greater freedom of speech on web 2.0 correlates with dominance of views linking vaccines to autism. Vaccine 2015 Mar 17;33(12):1422-1425. [doi: 10.1016/j.vaccine.2015.01.078] [Medline: 25665960]