Design feasibility of an automated, machine-learning based feedback system for motivational interviewing

Psychotherapy

Volumn 56, Issue 2, 2019, Pages 318-328

  Imel, Zac E ^a   Pace, Brian T ^a   Soma, Christina S ^a   Tanana, Michael ^a   Hirsch, Tad ^b   Gibson, James ^c   Georgiou, Panayiotis ^c   Narayanan, Shrikanth ^c   Atkins, David C ^d  

^a UNIVERSITY OF UTAH   (United States)

^b NORTHEASTERN UNIVERSITY   (United States)

^c University of Southern California   (United States)

^d UNIVERSITY OF WASHINGTON   (United States)

Author keywords

Adherence; Dissemination; Feedback; Machine learning; Motivational interviewing

Indexed keywords

ADULT; ARTICLE; CLINICAL ARTICLE; CONTROLLED STUDY; EMPATHY; FEASIBILITY STUDY; FEEDBACK SYSTEM; FEMALE; GENETIC TRANSCRIPTION; HUMAN; MACHINE LEARNING; MALE; MOTIVATIONAL INTERVIEWING; PERCEPTION; PROOF OF CONCEPT; PSYCHOTHERAPY; SATISFACTION; CLINICAL COMPETENCE; MENTAL DISEASE; PROCEDURES; PSYCHOLOGICAL FEEDBACK; PSYCHOLOGY;

ADULT; CLINICAL COMPETENCE; FEASIBILITY STUDIES; FEEDBACK, PSYCHOLOGICAL; FEMALE; HUMANS; MACHINE LEARNING; MALE; MENTAL DISORDERS; MOTIVATIONAL INTERVIEWING;

EID: 85063985695     PISSN: 00333204     EISSN: 19391536     Source Type: Journal    
DOI: 10.1037/pst0000221     Document Type: Article

References (47)

1. Atkins D. C.,Steyvers M.,Imel Z. E.,Smyth P. , 2014,Scaling up the evaluation of psychotherapy: Evaluating motivational interviewing fidelity via statistical text classification.Implementation Science, 9,49, http://dx.doi.org/10.1186/1748-5908-9-49
2. Baer, J. S., Rosengren, D. B., Dunn, C. W., Wells, E. A., Ogle, R. L., Hartzler, B. (2004). An evaluation of workshop training in motivational interviewing for addiction and mental health clinicians. Drug and Alcohol Dependence, 73, 99-106. http://dx.doi.org/10.1016/j.drugalcdep.2003.10.001
3. Baer, J. S., Wells, E. A., Rosengren, D. B., Hartzler, B., Beadnell, B., Dunn, C. (2009). Agency context and tailored training in technology transfer: A pilot evaluation of motivational interviewing training for community counselors. Journal of Substance Abuse Treatment, 37, 191-202. http://dx.doi.org/10.1016/j.jsat.2009.01.003
4. Bauer, A. M., Baldwin, S. A., Anguera, J. A., Areán, P. A., Atkins, D. C. (2018). Comparing approaches to mobile depression assessment for measurement-based care. Journal of Medical Internet Research, 20, e10001. http://dx.doi.org/10.2196/10001
5. Berger, A. L., Pietra, V. J. D., Pietra, S. A. D. (1996). A maximum entropy approach to natural language processing. Computational Linguistics, 22, 39-71.
6. Can, D., Marín, R. A., Georgiou, P. G., Imel, Z. E., Atkins, D. C., Narayanan, S. S. (2016). "It sounds like...": A natural language processing approach to detecting counselor reflections in motivational interviewing. Journal of Counseling Psychology, 63, 343-350. http://dx.doi.org/10.1037/cou0000111
7. Creed, T. A., Frankel, S. A., German, R. E., Green, K. L., Jager-Hyman, S., Taylor, K. P.,... Beck, A. T. (2016). Implementation of transdiagnostic cognitive therapy in community behavioral health: The Beck Community Initiative. Journal of Consulting and Clinical Psychology, 84, 1116-1126. http://dx.doi.org/10.1037/ccp0000105
8. Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55, 78-87. http://dx.doi.org/10.1145/2347736.2347755
9. Drucker, H., Burges, C. J. C., Kaufman, L., Smola, A. J., Vapnik, V. (1997). Support vector regression machines. In M. C. Mozer, M. I. Jordan, & T. Petsche (Eds.), Advances in neural information processing systems (Vol. 9, pp. 155-161). Cambridge, MA: MIT Press. Retrieved from http://papers.nips.cc/paper/1238-support-vector-regressionmachines. pdf
10. Epstein, M. L., Epstein, B. B., Brosvic, G. M. (2001). Immediate feedback during academic testing. Psychological Reports, 88, 889-894. http://dx.doi.org/10.2466/pr0.2001.88.3.889
11. Gaut, G., Steyvers, M., Imel, Z. E., Atkins, D. C., Smyth, P. (2017). Content coding of psychotherapy transcripts using labeled topic models. IEEE Journal of Biomedical and Health Informatics, 21, 476-487. http://dx.doi.org/10.1109/JBHI.2015.2503985
12. Gibson, J., Gray, G., Hirsch, T., Imel, Z. E., Narayanan, S. S., Atkins, D. S. (2016, June). Developing an automated report card for addiction counseling: The counselor observer ratings expert for MI (CORE-MI). Paper presented at CHI, San Jose, CA.
13. Hasan, M., Kotov, A., Carcone, A. D., Dong, M., Naar, S., Hartlieb, K. B. (2016). A study of the effectiveness of machine learning methods for classification of clinical interview fragments into a large number of categories. Journal of Biomedical Informatics, 62, 21-31. http://dx.doi.org/10.1016/j.jbi.2016.05.004
14. Hattie, J., Timperley, H. (2007). The power of feedback. Review of Educational Research, 77, 81-112. http://dx.doi.org/10.3102/003465430298487
15. Hirsch, T., Merced, K., Narayanan, S., Imel, Z. E., Atkins, D. C. (2017, June). Designing contestability: Interaction design, machine learning, and mental health. Paper presented at DIS: Designing Interactive Systems. Edinburgh, United Kingdom. http://dx.doi.org/10.1145/3064663.3064703
16. Hirsch, T., Soma, C., Merced, K., Kuo, P., Dembe, A., Caperton, D.,... Imel, Z. E. (2018, June). "It's hard to argue with a computer:" Investigating Psychotherapists' Attitudes towards Automated Evaluation. Paper presented at ACM Conference on Designing Interactive Systems (DIS'18), Hong Kong.
17. Hirschberg, J., Manning, C. D. (2015). Advances in natural language processing. Science, 349, 261-266. http://dx.doi.org/10.1126/science.aaa8685
18. Howes, C., Purver, M., McCabe, R. (2013, March). Investigating topic modelling for therapy dialogue analysis. Proceedings of the Computational Semantics in Clinical Text workshop at IWCS Potsdam. Baltimore, Maryland: Association for Computational Linguistics. Retrieved from https://pdfs.semanticscholar.org/8726/b8e4afb77598eb37ef 444f9f6bc245ce865f.pdf
19. Howes, C., Purver, M., McCabe, R. (2014). Linguistic indicators of severity and progress in online text-based therapy for depression. Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality (pp. 7-16), Baltimore, MD. Retrieved from https://www.aclweb.org/anthology/W/W14/W14-3202.pdf
20. Imel, Z. E., Steyvers, M., Atkins, D. C. (2015). Computational psychotherapy research: Scaling up the evaluation of patient-provider interactions. Psychotherapy, 52, 19-30. http://dx.doi.org/10.1037/a0036841
21. Jones, K. S. (1994). Natural language processing: A historical review. In A. Zampolli, N. Calzolari, & M. Palmer (Eds.), Current issues in computational linguistics: In Honour of Don Walker (pp. 3-16). Dordrecht, the Netherlands: Springer. http://dx.doi.org/10.1007/978-0-585-35958-8-1
22. Jurafsky, D., Martin, J. H. (2008). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition (2nd ed.). Upper Saddle River, NJ: Prentice Hall.
23. Kendrick, T., El-Gohary, M., Stuart, B., Gilbody, S., Churchill, R., Aiken, L.,... Moore, M. (2016). Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. Cochrane Database of Systematic Reviews, 7, CD011119. http://dx.doi.org/10.1002/14651858.CD011119.pub2
24. Kluger, A. N., DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119, 254-284. http://dx.doi.org/10.1037/0033-2909.119.2.254
25. Kulik, J. A., Kulik, C. L. C. (1988). Timing of feedback and verballearning. Review of Educational Research, 58, 79-97. http://dx.doi.org/10.3102/00346543058001079
26. Ladany, N., Hill, C. E., Corbett, M. M., Nutt, E. A. (1996). Nature, extent, and importance of what psychotherapy trainees do not disclose to their supervisors. Journal of Counseling Psychology, 43, 10-24. http://dx.doi.org/10.1037/0022-0167.43.1.10
27. Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., Hawkins, E. J. (2005). Providing feedback to psychotherapists on their patients' progress: Clinical results and practice suggestions. Journal of Clinical Psychology, 61, 165-174. http://dx.doi.org/10.1002/jclp.20113
28. Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L., Goates, M. (2002). Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clinical Psychology and Psychotherapy, 9, 91-103. http://dx.doi.org/10.1002/cpp.324
29. Lord, S. P., Can, D., Yi, M., Marin, R., Dunn, C. W., Imel, Z. E.,... Atkins, D. C. (2015). Advancing methods for reliably assessing motivational interviewing fidelity using the motivational interviewing skills code. Journal of Substance Abuse Treatment, 49, 50-57. http://dx.doi.org/10.1016/j.jsat.2014.08.005
30. Lundahl, B. W., Kunz, C., Brownell, C., Tollefson, D., Burke, B. L. (2010). A meta-analysis of motivational interviewing: Twenty-five years of empirical studies. Research on Social Work Practice, 20, 137-160. http://dx.doi.org/10.1177/1049731509347850
31. McHugh, R. K., Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments. A review of current efforts. American Psychologist, 65, 73-84. http://dx.doi.org/10.1037/a0018121
32. Miller, W. R., Moyers, T. B., Ernst, D., Amrhein, P. (2008). Manual for the Motivational Interviewing Skill Code (MISC), Version 2.1. Albuquerque: Center on Alcoholism. Substance Abuse, and Addictions, The University of New Mexico.
33. Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J., Pirritano, M. (2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology, 72, 1050-1062. http://dx.doi.org/10.1037/0022-006X.72.6.1050
34. Moyers, T. B., Martin, T., Manuel, J. K., Miller, W. R., Ernst, D. (2010). Revised global scales: Motivational interviewing treatment integrity 3.1. 1 (MITI 3.1. 1). Unpublished manuscript, University of New Mexico, Albuquerque, NM. Retrieved from http://www.marrch.org/associations/4671/files/02-09-MITI-3-1-1.pdf
35. Norman, D. (2013). The design of everyday things: Revised and expanded edition. New York, NY: Basic Books. Retrieved from https://market.android.com/detailsid-book-qBfRDQAAQBAJ
36. Olfson, M., Marcus, S. C. (2010). National trends in outpatient psychotherapy. The American Journal of Psychiatry, 167, 1456-1463. http://dx.doi.org/10.1176/appi.ajp.2010.10040570
37. Pace, B., Tanana, M., Xiao, B., Dembe, A., Soma, C., Steyvers, M., Imel, Z. E. (2016). What about the words Natural language processing in psychotherapy. Psychotherapy Bulletin, 51, 17-18.
38. Pérez-Rosas, V., Mihalcea, R., Resnicow, K., Singh, S., Ann, L., Goggin, K. J., Catley, D. (2017, April). Predicting counselor behaviors in motivational interviewing encounters. Paper presented at the 15th Conference of the European Chapter of the Association for Computational Linguistics: Vol. 1, Long Papers. Retrieved from http://www.aclweb.org/anthology/E17-1106. http://dx.doi.org/10.18653/v1/E17-1106
39. Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N.,... Vesely, K. (2011, December). The kaldi speech recognition toolkit. Paper presented at IEEE Signal Processing Society. Retrieved from https://infoscience.epfl.ch/record/192584
40. Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36, 24-34. http://dx.doi.org/10.1007/s10488-008-0197-4
41. Rogers, C. R. (1951). Studies in client-centered psychotherapy III: The case of Mrs. Oak-A research analysis. Psychological Service Center Journal, 3, 47-165.
42. Rush, A. J. (2015). Isn't it about time to employ measurement-based care in practice. The American Journal of Psychiatry, 172, 934-936. http://dx.doi.org/10.1176/appi.ajp.2015.15070928
43. Schwalbe, C. S., Oh, H. Y., Zweben, A. (2014). Sustaining motivational interviewing: A meta-analysis of training studies. Addiction, 109, 1287-1294. http://dx.doi.org/10.1111/add.12558
44. Tanana, M. J., Soma, C. S., Srikumar, V., Atkins, D. C., Imel, Z. E. (2018). Development and evaluation of ClientBot: A patient-like conversational agent to train basic counseling skills. Journal of Medical Internet Research. Advance online publication. http://dx.doi.org/10.2196/preprints.12529
45. Tanana, M., Hallgren, K. A., Imel, Z. E., Atkins, D. C., Srikumar, V. (2016). A comparison of natural language processing methods for automated coding of motivational interviewing. Journal of Substance Abuse Treatment, 65, 43-50. http://dx.doi.org/10.1016/j.jsat.2016.01.006
46. Xiao, B., Huang, C., Imel, Z. E., Atkins, D. C., Georgiou, P., Narayanan, S. S. (2016). A technology prototype system for rating therapist empathy from audio recordings in addiction counseling. PeerJ Computer Science, 2, e59. http://dx.doi.org/10.7717/peerj-cs.59
47. Xiao, B., Imel, Z. E., Georgiou, P. G., Atkins, D. C., Narayanan, S. S. (2015). Rate my therapist": Automated detection of empathy in drug and alcohol counseling via speech and language processing. PLoS ONE, 10, e0143055. http://dx.doi.org/10.1371/journal.pone.0143055