Interrater reliability of mhealth app rating measures: Analysis of top depression and smoking cessation apps

JMIR mHealth and uHealth

Volumn 4, Issue 1, 2016, Pages

  Powell, Adam C ^a   Torous, John ^b,c   Chan, Steven ^d   Raynor, Geoffrey Stephen ^b,c   Shwarts, Erik ^d   Shanahan, Meghan ^c   Landman, Adam B ^c  

^a Payer + Provider Syndicate   (United States)

^b Harvard Longwood Psychiatry Residency Training Program   (United States)

^c HARVARD MEDICAL SCHOOL   (United States)

^d UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Evaluation studies; Health apps; Mental health; Mobile applications; Ratings

Indexed keywords

EID: 84999858539     PISSN: None     EISSN: 22915222     Source Type: Journal    
DOI: 10.2196/mhealth.5176     Document Type: Article

References (40)

1. IMS Health. 2015. IMS Institute for Healthcare EconomicsURL:http://www.imshealth.com/portal/site/imshealth/ menuitem.0be132395225d98ee566e5661ad8c22a/?vgnextoid=a64de5fda6370410VgnVCM10000076192ca2RCRD [accessed 2015-09-24] [WebCite Cache ID 6bmuQEZsG]
2. Powell AC, Landman AB, Bates DW. In search of a few good apps. JAMA 2014 May 14;311(18):1851-1852. [doi: 10.1001/jama.2014.2564] [Medline: 24664278]
3. Powell AC, Landman AB, Bates DW. Certification of mobile apps for health care - reply. JAMA 2014 Sep 17;312(11):1156-1157. [doi: 10.1001/jama.2014.9007] [Medline: 25226488]
4. Shen N, Levitan M, Johnson A, Bender JL, Hamilton-Page M, Jadad Alejandro AR, et al. Finding a depression app: a review and content analysis of the depression app marketplace. JMIR Mhealth Uhealth 2015;3(1):e16 [FREE Full text] [doi: 10.2196/mhealth.3713] [Medline: 25689790]
5. Torous J, Powell AC. Current research and trends in the use of smartphone applications for mood disorders. Internet Interventions 2015 May;2(2):169-173. [doi: 10.1016/j.invent.2015.03.002]
6. Food and Drug Administration. 2015 Feb 09. Mobile Medical Applications: Guidance for Industry and Food and Drug Administration StaffURL:http://www.fda.gov/downloads/MedicalDevices/.../UCM263366.pdf [accessed 2015-11-01] [WebCite Cache ID 6ek8YGpuF]
7. Knapton S. The Telegraph. 2015 Oct 12. Most NHS depression apps are unproven, warn health experts InternetURL:http:/ /www.telegraph.co.uk/news/nhs/11926616/Most-NHS-depression-apps-are-unproven-warn-health-experts.html [accessed 2015-10-24] [WebCite Cache ID 6cWLH90jy]
8. Wicks P, Chiauzzi E. 'Trust but verify' - five approaches to ensure safe medical apps. BMC Med 2015;13:205 [FREE Full text] [doi: 10.1186/s12916-015-0451-z] [Medline: 26404791]
9. Boudreaux ED, Waring ME, Hayes RB, Sadasivam RS, Mullen S, Pagoto S. Evaluating and selecting mobile health apps: strategies for healthcare providers and healthcare organizations. Transl Behav Med 2014 Dec;4(4):363-371 [FREE Full text] [doi: 10.1007/s13142-014-0293-9] [Medline: 25584085]
10. Dayer L, Heldenbrand S, Anderson P, Gubbins PO, Martin BC. Smartphone medication adherence apps: potential benefits to patients and providers. J Am Pharm Assoc (2003) 2013;53(2):172-181 [FREE Full text] [doi: 10.1331/JAPhA.2013.12202] [Medline: 23571625]
11. Ostrovsky A, Deen N, Simon A, Mate K. Institute for Healthcare Improvement. Cambridge, MA; 2014 Jun. A framework for selecting digital health technology: IHI Innovation ReportURL:http://www.ihi.org/resources/Pages/Publications/ AFrameworkforSelectingDigitalHealthTechnology.aspx [accessed 2016-01-23] [WebCite Cache ID 6ek9vtCCA]
12. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015;3(1):e27 [FREE Full text] [doi: 10.2196/mhealth.3422] [Medline: 25760773]
13. Chan S, Torous J, Hinton L, Yellowlees P. Towards a Framework for Evaluating Mobile Mental Health Apps. Telemed J E Health 2015 Dec;21(12):1038-1041. [doi: 10.1089/tmj.2015.0002] [Medline: 26171663]
14. Torous J, Powell AC. Current research and trends in the use of smartphone applications for mood disorders. Internet Interventions 2015 May;2(2):169-173. [doi: 10.1016/j.invent.2015.03.002]
15. Milani P, Coccetta CA, Rabini A, Sciarra T, Massazza G, Ferriero G. Mobile smartphone applications for body position measurement in rehabilitation: a review of goniometric tools. PM R 2014 Nov;6(11):1038-1043. [doi: 10.1016/j.pmrj.2014.05.003] [Medline: 24844445]
16. Gajecki M, Berman AH, Sinadinovic K, Rosendahl I, Andersson C. Mobile phone brief intervention applications for risky alcohol use among university students: a randomized controlled study. Addict Sci Clin Pract 2014;9:11 [FREE Full text] [doi: 10.1186/1940-0640-9-11] [Medline: 24985342]
17. Torous J, Keshavan M, Gutheil T. Promise and perils of digital psychiatry. Asian J Psychiatr 2014 Aug;10:120-122. [doi: 10.1016/j.ajp.2014.06.006] [Medline: 25042968]
18. West JH, Hall PC, Hanson CL, Barnes MD, Giraud-Carrier C, Barrett J. There's an app for that: content analysis of paid health and fitness apps. J Med Internet Res 2012;14(3):e72 [FREE Full text] [doi: 10.2196/jmir.1977] [Medline: 22584372]
19. BinDhim NF, Hawkey A, Trevena L. A systematic review of quality assessment methods for smartphone health apps. Telemed J E Health 2015 Feb;21(2):97-104. [doi: 10.1089/tmj.2014.0088] [Medline: 25469795]
20. Abroms LC, Lee WJ, Bontemps-Jones J, Ramani R, Mellerson J. A content analysis of popular smartphone apps for smoking cessation. Am J Prev Med 2013 Dec;45(6):732-736 [FREE Full text] [doi: 10.1016/j.amepre.2013.07.008] [Medline: 24237915]
21. Choi J, Noh G, Park D. Smoking cessation apps for smartphones: content analysis with the self-determination theory. J Med Internet Res 2014;16(2):e44 [FREE Full text] [doi: 10.2196/jmir.3061] [Medline: 24521881]
22. Shen N, Levitan M, Johnson A, Bender JL, Hamilton-Page M, Jadad AA, et al. Finding a depression app: a review and content analysis of the depression app marketplace. JMIR Mhealth Uhealth 2015;3(1):e16 [FREE Full text] [doi: 10.2196/mhealth.3713] [Medline: 25689790]
23. National Institute of Mental Health. 2015. Major Depression Among AdultsURL:http://www.nimh.nih.gov/health/statistics/ prevalence/major-depression-among-adults.shtml [accessed 2015-10-24] [WebCite Cache ID 6cWKbu31I]
24. Centers for Disease Control and Prevention. 2015 Aug 25. Smoking & Tobacco UseURL:http://www.cdc.gov/tobacco/ data_statistics/fact_sheets/adult_data/cig_smoking/ [accessed 2015-10-24] [WebCite Cache ID 6cWKj9etb]
25. IMS Institute for Healthcare Informatics. 2015. Patient Adoption of mHealthURL:http://www.imshealth.com/portal/site/ imshealth/menuitem.762a961826aad98f53c753c71ad8c22a/ ?vgnextoid=e0f913850c8b1410VgnVCM10000076192ca2RCRD&vgnextchannel=736de5fda6370410VgnVCM10000076192ca2RCRD&vgnextfmt=default [accessed 2015-10-24] [WebCite Cache ID 6cWM6CATH]
26. Donker T, Petrie K, Proudfoot J, Clarke J, Birch M, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res 2013;15(11):e247 [FREE Full text] [doi: 10.2196/jmir.2791] [Medline: 24240579]
27. Anxiety and Depression Association of America. 2015. Mobile AppsURL:http://www.adaa.org/finding-help/mobile-apps [accessed 2015-09-24] [WebCite Cache ID 6bmudjDY7]
28. PsyberGuide. 2015. PsyberGuide ratings - explanationURL:http://psyberguide.org/psyberguide-ratings-explanation/ [accessed 2015-09-24] [WebCite Cache ID 6bmujJ5ug]
29. Kharrazi H, Chisholm R, VanNasdale D, Thompson B. Mobile personal health records: an evaluation of features and functionality. Int J Med Inform 2012 Sep;81(9):579-593. [doi: 10.1016/j.ijmedinf.2012.04.007] [Medline: 22809779]
30. Pandey A, Hasan S, Dubey D, Sarangi S. Smartphone apps as a source of cancer information: changing trends in health information-seeking behavior. J Cancer Educ 2013 Mar;28(1):138-142. [doi: 10.1007/s13187-012-0446-9] [Medline: 23275239]
31. Lewis TL. A systematic self-certification model for mobile medical apps. J Med Internet Res 2013;15(4):e89 [FREE Full text] [doi: 10.2196/jmir.2446] [Medline: 23615332]
32. Martínez-Pérez B, de la Torre-Díez I, Candelas-Plasencia S, López-Coronado M. Development and evaluation of tools for measuring the quality of experience (QoE) in mHealth applications. J Med Syst 2013 Oct;37(5):9976. [doi: 10.1007/s10916-013-9976-x] [Medline: 24014255]
33. Ferrero-Alvarez-Rementeria J, Santana-Lopez V, Escobar-Ubreva A, Vazquez-Vazquez M. European Journal of ePractice. 2013 Dec. Quality and safety strategy for mobile health applications: a certification programmeURL:https://joinup. ec.europa.eu/sites/default/files/28/06/c2/ePractice-Journal-Vol.21-December%202013.pdf [accessed 2016-01-29] [WebCite Cache ID 6etzyfH47]
34. Krippendorff K. Content analysis: an introduction to its methodology. Thousand Oaks, CA: Sage; 2004.
35. Krippendorff K. Reliability in content analysis. In: Content analysis: an introduction to its methodology. Thousand Oaks, CA: Sage; 2004:411-433.
36. Schaer P. Better than their reputation? On the reliability of relvance assessments with students. In: Information Access Evaluation Multilinguality, Multimodality, and Visual Analytics Lecture Notes in Computer Science. Berlin: Springer; 2012:7488-7135.
37. Artstein R, Poesio M. Inter-Coder Agreement for Computational Linguistics. Comput Ling 2008 Dec;34(4):555-596. [doi: 10.1162/coli.07-034-R2]
38. Staudt A, Krewel M. IDEAS Statistical Software Components. Boston College: Research Papers in Economics; 2015 Jan 24. Stata module to compute Krippendorff's alpha intercoder reliability coefficientURL:https://ideas.repec.org/c/boc/bocode/ s457750.html [accessed 2016-01-23] [WebCite Cache ID 6ekBxB93W]
39. Usability.gov. System Usability Scale (SUS)URL:http://www.usability.gov/how-to-and-tools/methods/system-usability-scale. html [accessed 2015-09-24] [WebCite Cache ID 6bmxQC8U4]
40. Austin JM, Jha AK, Romano PS, Singer SJ, Vogus TJ, Wachter RM, et al. National hospital ratings systems share few common scores and may generate confusion instead of clarity. Health Aff (Millwood) 2015 Mar;34(3):423-430. [doi: 10.1377/hlthaff.2014.0201] [Medline: 25732492]