Enlight: A comprehensive quality and therapeutic potential evaluation tool for mobile and web-based eHealth interventions

Journal of Medical Internet Research

Volumn 19, Issue 3, 2017, Pages

  Baumel, Amit ^a,b   Faber, Keren ^a   Mathur, Nandita ^a   Kane, John M ^a,b   Muench, Fred ^a,b  

^a FEINSTEIN INSTITUTE FOR MEDICAL RESEARCH   (United States)

^b DONALD AND BARBARA ZUCKER SCHOOL OF MEDICINE AT HOFSTRA NORTHWELL   (United States)

Author keywords

Assessment; Behavior change; EHealth; Evaluation; MHealth; Persuasive design; Quality; Therapeutic alliance

Indexed keywords

BEHAVIOR CHANGE; CHECKLIST; CONCURRENT VALIDITY; CRONBACH ALPHA COEFFICIENT; EVIDENCE BASED PRACTICE CENTER; HUMAN; INTERNAL CONSISTENCY; INTERRATER RELIABILITY; MENTAL HEALTH; MOBILE APPLICATION; PRIVACY; PROBABILITY; QUALITY CONTROL; STATISTICAL MODEL; SYSTEMATIC REVIEW; TELEHEALTH; INTERNET; PROCEDURES; TELEMEDICINE;

HUMANS; INTERNET; MOBILE APPLICATIONS; TELEMEDICINE;

EID: 85016403057     PISSN: None     EISSN: 14388871     Source Type: Journal    
DOI: 10.2196/jmir.7270     Document Type: Article

References (62)

1. Aitken M, Gauntlett C. Patient Apps for Improved Healthcare from Novelty to Mainstream. Parsippany, NJ: IMS Institute for Healthcare Informatics; 2013.
2. Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, et al. Mobile health technology evaluation: The mHealth evidence workshop. Am J Prev Med 2013 Aug;45(2):228-236 [FREE Full text] [doi: 10.1016/j.amepre.2013.03.017] [Medline: 23867031]
3. Baumel A. Making the case for a feasible evaluation method of available e-mental health products. Adm Policy Ment Health 2016 Sep 12:1-4. [doi: 10.1007/s10488-016-0764-z] [Medline: 27620191]
4. Baumel A, Muench F. Heuristic evaluation of eHealth interventions: Establishing standards that relate to the therapeutic process perspective. JMIR Ment Health 2016 Jan 13;3(1):e5 [FREE Full text] [doi: 10.2196/mental.4563] [Medline: 26764209]
5. Hekler EB, Klasnja P, Riley WT, Buman MP, Huberty J, Rivera DE, et al. Agile science: Creating useful products for behavior change in the real world. Transl Behav Med 2016 Jun;6(2):317-328 [FREE Full text] [doi: 10.1007/s13142-016-0395-7] [Medline: 27357001]
6. Patrick K, Hekler E, Estrin D, Mohr DC, Riper H, Crane D, et al. The pace of technologic change: Implications for digital health behavior intervention research. Am J Prev Med 2016 Nov;51(5):816-824. [doi: 10.1016/j.amepre.2016.05.001] [Medline: 27745681]
7. Powell AC, Torous J, Chan S, Raynor GS, Shwarts E, Shanahan M, et al. Interrater reliability of mHealth app rating measures: Analysis of top depression and smoking cessation apps. JMIR Mhealth Uhealth 2016;4(1):e15 [FREE Full text] [doi: 10.2196/mhealth.5176] [Medline: 26863986]
8. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: An instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999 Feb;53(2):105-111 [FREE Full text] [Medline: 10396471]
9. Olsina L, Rossi G. Measuring web application quality with WebQEM. IEEE Multimedia 2002;9(4):20-29.
10. Provost M, Koompalum D, Dong D, Martin BC. The initial development of the WebMedQual scale: Domain assessment of the construct of quality of health web sites. Int J Med Inform 2006 Jan;75(1):42-57. [doi: 10.1016/j.ijmedinf.2005.07.034] [Medline: 16169770]
11. Stoyanov S, Hides L, Kavanagh D, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: A new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar 11;3(1):e27 [FREE Full text] [doi: 10.2196/mhealth.3422] [Medline: 25760773]
12. PsyberGuide. 2015. URL: Http://psyberguide.org/ [accessed 2015-04-10] [WebCite Cache ID 6Xh99eQmZ]
13. Anxiety and Depression Association of America. 2016. Mental health apps URL: Https://www.adaa.org/finding-help/ mobile-apps [accessed 2016-11-02] [WebCite Cache ID 6lhtKNvkc]
14. JMIR Publications. 2015. Contribute to mHealth research and development of the new JMIR mHealth peer-review tool for mobile apps! URL: Http://www.jmir.org/announcement/view/77 [accessed 2015-08-10] [WebCite Cache ID 6agJfjxOb]
15. Fogg B. Persuasive Technology: Using Computers to Change What We Think and Do (Interactive Technologies). San Francisco, CA: Morgan Kaufmann; 2002.
16. Persuasion in Design. 2017. URL: Https://uxmag.com/articles/persuasion-in-design [accessed 2017-02-09] [WebCite Cache ID 6o9P2xMjD]
17. Kientz J, Choe E, Birch B. Heuristic evaluation of persuasive health technologies. 2010 Presented at: 1st ACM International Health Informatics Symposium; Nov 11-12, 2010; Arlington, VA.
18. Nielsen J, Molich R. Heuristic evaluation of user interfaces. 1990 Presented at: CHI '90 SIGCHI Conference on Human Factors in Computing Systems; Apr 1-5, 1990; Seattle, WA.
19. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE. Persuasive system design does matter: A systematic review of adherence to web-based interventions. J Med Internet Res 2012;14(6):e152 [FREE Full text] [doi: 10.2196/jmir.2104] [Medline: 23151820]
20. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: A systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010;12(1):e4 [FREE Full text] [doi: 10.2196/jmir.1376] [Medline: 20164043]
21. Martin DJ, Garske JP, Davis MK. Relation of the therapeutic alliance with outcome and other variables: A meta-analytic review. J Consult Clin Psychol 2000 Jun;68(3):438-450. [Medline: 10883561]
22. Kazdin AE, Marciano PL, Whitley MK. The therapeutic alliance in cognitive-behavioral treatment of children referred for oppositional, aggressive, and antisocial behavior. J Consult Clin Psychol 2005 Aug;73(4):726-730. [doi: 10.1037/0022-006X.73.4.726] [Medline: 16173860]
23. Klein DN, Schwartz JE, Santiago NJ, Vivian D, Vocisano C, Castonguay LG, et al. Therapeutic alliance in depression treatment: Controlling for prior change and patient characteristics. J Consult Clin Psychol 2003 Dec;71(6):997-1006. [doi: 10.1037/0022-006X.71.6.997] [Medline: 14622075]
24. Ferreira PH, Ferreira ML, Maher CG, Refshauge KM, Latimer J, Adams RD. The therapeutic alliance between clinicians and patients predicts outcome in chronic low back pain. Phys Ther 2013 Apr;93(4):470-478. [doi: 10.2522/ptj.20120137] [Medline: 23139428]
25. Kim SC, Kim S, Boren D. The quality of therapeutic alliance between patient and provider predicts general satisfaction. Mil Med 2008 Jan;173(1):85-90. [Medline: 18251337]
26. Bennett JK, Fuertes JN, Keitel M, Phillips R. The role of patient attachment and working alliance on patient adherence, satisfaction, and health-related quality of life in lupus treatment. Patient Educ Couns 2011 Oct;85(1):53-59. [doi: 10.1016/j.pec.2010.08.005] [Medline: 20869188]
27. Molassiotis A, Morris K, Trueman I. The importance of the patient-clinician relationship in adherence to antiretroviral medication. Int J Nurs Pract 2007 Dec;13(6):370-376. [doi: 10.1111/j.1440-172X.2007.00652.x] [Medline: 18021166]
28. Bickmore T, Gruber A, Picard R. Establishing the computer-patient working alliance in automated health behavior change interventions. Patient Educ Couns 2005 Oct;59(1):21-30. [doi: 10.1016/j.pec.2004.09.008] [Medline: 16198215]
29. Clarke J, Proudfoot J, Whitton A, Birch M, Boyd M, Parker G, et al. Therapeutic alliance with a fully automated mobile phone and web-based intervention: Secondary analysis of a randomized controlled trial. JMIR Ment Health 2016 Feb 25;3(1):e10 [FREE Full text] [doi: 10.2196/mental.4656] [Medline: 26917096]
30. Bergman NL, Carlbring P, Linna E, Andersson G. Role of the working alliance on treatment outcome in tailored internet-based cognitive behavioural therapy for anxiety disorders: Randomized controlled pilot trial. JMIR Res Protoc 2013;2(1):e4 [FREE Full text] [doi: 10.2196/resprot.2292] [Medline: 23612437]
31. Cavanagh K, Millings A. (Inter) personal computing: The role of the therapeutic relationship in e-mental health. J Contemp Psychother 2013;43(4):197-206.
32. Holter MT, Johansen A, Brendryen H. How a fully automated eHealth program simulates three therapeutic processes: A case study. J Med Internet Res 2016 Jun 28;18(6):e176 [FREE Full text] [doi: 10.2196/jmir.5415] [Medline: 27354373]
33. Barazzone N, Cavanagh K, Richards DA. Computerized cognitive behavioural therapy and the therapeutic alliance: A qualitative enquiry. Br J Clin Psychol 2012 Nov;51(4):396-417. [doi: 10.1111/j.2044-8260.2012.02035.x] [Medline: 23078210]
34. Eysenbach G, Powell J, Kuss O, Sa E. Empirical studies assessing the quality of health information for consumers on the world wide web: A systematic review. JAMA 2002;287(20):2691-2700. [Medline: 12020305]
35. Gomez QS, Walthouwer MJ, Schulz DN, de VH. mHealth or eHealth? Efficacy, use, and appreciation of a web-based computer-tailored physical activity intervention for dutch adults: A randomized controlled trial. J Med Internet Res 2016 Nov 09;18(11):e278 [FREE Full text] [doi: 10.2196/jmir.6171] [Medline: 27829576]
36. Baumel A, Pawar A, Mathur N, Kane J, Corell C. Technology assisted parent training programs for children and adolescents with disruptive behaviors: A systematic review. J Clin Psychiatry 2017:A (forthcoming).
37. Nielsen J. Enhancing the explanatory power of usability heuristics. 1994 Presented at: SIGCHI Conference on Human Factors in Computing Systems; Apr 24-28, 1994; Boston, MA.
38. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Ann Intern Med 2009 Aug 18;151(4):W65-W94. [Medline: 19622512]
39. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006 Jan;3(2):77-101. [doi: 10.1191/1478088706qp063oa]
40. Mokdad AH, Marks JS, Stroup DF, Gerberding JL. Actual causes of death in the United States, 2000. JAMA 2004 Mar 10;291(10):1238-1245. [doi: 10.1001/jama.291.10.1238] [Medline: 15010446]
41. Van Deursen AJ, Van Dijk JA. Using the Internet: Skill related problems in users' online behavior. Interact Comput 2009;21(5):393-402.
42. iProspect. Search engine user behavior study. 2006. URL: Http://district4.extension.ifas.ufl.edu/Tech/TechPubs/ WhitePaper-2006-SearchEngineUserBehavior.pdf [WebCite Cache ID 6p0pvxfeq]
43. RANDOM.ORG. 2016. List randomizer URL: Https://www.random.org/lists/ [accessed 2016-12-22] [WebCite Cache ID 6mwaCaO2t]
44. Westen D, Lohr N. Social Cognition and Object Relations Scale (SCORS): Manual for Coding TAT Data. Ann Arbor, MI: Department of Psychology, University of Michigan; 1990.
45. Blatt S, Chevron E, Quinlan D, Schaffer C, Wein S. The Assessment of Qualitative and Structural Dimensions of Object Representations (revised edition). New Haven, CT: Yale University; 1988.
46. Cramer P. The Development of Defense Mechanisms: Theory, Research, and Assessment. New York: Springer Science & Business Media; 2012.
47. Newcombe RG. Confidence Intervals for Proportions and Related Measures of Effect Size. Boca Raton, FL: CRC Press; 2012.
48. Zou GY. Sample size formulas for estimating intraclass correlation coefficients with precision and assurance. Stat Med 2012 Dec 20;31(29):3972-3981. [doi: 10.1002/sim.5466] [Medline: 22764084]
49. Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull 1979 Mar;86(2):420-428. [Medline: 18839484]
50. Bartko JJ. The intraclass correlation coefficient as a measure of reliability. Psychol Rep 1966 Aug;19(1):3-11. [doi: 10.2466/pr0.1966.19.1.3] [Medline: 5942109]
51. Cronbach L. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16(3):297-334.
52. Streiner DL. Being inconsistent about consistency: when coefficient alpha does and doesn't matter. J Pers Assess 2003 Jun;80(3):217-222. [doi: 10.1207/S15327752JPA8003-01] [Medline: 12763696]
53. Sim J, Wright CC. The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Phys Ther 2005 Mar;85(3):257-268 [FREE Full text] [Medline: 15733050]
54. Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J R Stat Soc Series B Stat Methodol 1995;57(1):289-300.
55. Gut A. Probability: A Graduate Course. Vol 75. New York: Springer Science & Business Media; 2012.
56. Alexa. 2017. URL: Http://www.alexa.com/ [accessed 2017-01-04] [WebCite Cache ID 6nGSLql6u]
57. Olteanu A, Peshterliev S, Liu X, Aberer K. Web credibility: Features exploration and credibility prediction. 2013 Presented at: 35th European conference on Advances in Information Retrieval; Mar 24-27, 2013; Moscow, Russia.
58. Similar Web. 2017. URL: Http://www.similarweb.com/ [accessed 2017-03-10] [WebCite Cache ID 6oruwtIA6]
59. O'Brien H, Toms E. What is user engagement? A conceptual framework for defining user engagement with technology. J Assoc Inf Sci Technol 2008;59(6):938-955.
60. Atkinson M, Kydd C. Individual characteristics associated with World Wide Web use: An empirical study of playfulness and motivation. SIGMIS Database 1997 Apr 01;28(2):53-62. [doi: 10.1145/264701.264705]
61. Althoff T, White RW, Horvitz E. Influence of Pokémon Go on physical activity: Study and implications. J Med Internet Res 2016 Dec 06;18(12):e315 [FREE Full text] [doi: 10.2196/jmir.6759] [Medline: 27923778]
62. Mohr DC, Cuijpers P, Lehman K. Supportive accountability: A model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res 2011;13(1):e30 [FREE Full text] [doi: 10.2196/jmir.1602] [Medline: 21393123]