Automatic audiovisual behavior descriptors for psychological disorder analysis

Image and Vision Computing

Volumn 32, Issue 10, 2014, Pages 648-658

  Scherer, Stefan ^a   Stratou, Giota ^a   Lucas, Gale ^a   Mahmoud, Marwa ^a,b   Boberg, Jill ^a   Gratch, Jonathan ^a   Rizzo, Albert ^a   Morency, Louis Philippe ^a  

^a University of Southern California   (United States)

^b UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Anxiety; Audiovisual; Automatic assessment; Depression; Nonverbal behavior; Post traumatic stress disorder; Psychological distress

Indexed keywords

AUDIOVISUAL; SURVEYS;

ANXIETY; AUTOMATIC ASSESSMENT; DEPRESSION; NONVERBAL BEHAVIOR; POST TRAUMATIC STRESS DISORDERS; PSYCHOLOGICAL DISTRESS;

BEHAVIORAL RESEARCH;

EID: 84906781906     PISSN: 02628856     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.imavis.2014.06.001     Document Type: Article

References (74)

1. T. Baltrusaitis, P. Robinson, and L.-P. Morency 3D constrained local model for rigid and non-rigid facial tracking IEEE Computer Vision and Pattern Recognition (CVPR 2012) 2012
2. L.-P. Morency, J. Whitehill, and J. Movellan Generalized adaptive view-based appearance model: integrated framework for monocular head pose estimation 8th IEEE International Conference on Automatic Face Gesture Recognition (FG08) 2008 1 8 10.1109/AFGR.2008.4813429
3. J. Whitehill, G. Littlewort, I. Fasel, M. Bartlett, and J. Movellan Toward practical smile detection IEEE Trans. Pattern Anal. Mach. Intell. 31 2009 2106 2111
4. S. Scherer, G. Stratou, M. Mahmoud, J. Boberg, J. Gratch, A. Rizzo, and L.-P. Morency Automatic behavior descriptors for psychological disorder analysis Proceedings of IEEE Conference on Automatic Face and Gesture Recognition, IEEE 2013
5. L.I. Reed, M. Sayette, and J.F. Cohn Impact of depression on response to comedy: a dynamic facial coding analysis J. Abnorm. Psychol. 116 2007 804 809
6. J.F. Cohn, T.S. Kruez, I. Matthews, Y. Ying, M.H. Nguyen, M.T. Padilla, F. Zhou, and F. De la Torre Detecting depression from facial actions and vocal prosody 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops 2009 1 7
7. J. Joshi, R. Goecke, G. Parker, and M. Breakspear Can body expressions contribute to automatic depression analysis? Proceedings of IEEE Conference on Automatic Face and Gesture Recognition 2013
8. N. Cummins, J.R. Epps, M.J. Breakspear, and R. Goecke An investigation of depressed speech detection: features and normalization Proceedings of Interspeech 2011, ISCA 2011
9. J. Joshi, R. Goecke, S. Alghowinem, A. Dhall, M. Wagner, J.R. Epps, G. Parker, and M. Breakspear Multimodal assistive technologies for depression diagnosis and monitoring Journal on Multimodal User Interfaces 7 2013 217 228
10. J. Joshi, A. Dhall, R. Goecke, and J.F. Cohn Relative body parts movement for automatic depression analysis Proceedings of Affective Computing and Intelligent Interaction (ACII), IEEE 2013 492 497
11. N. Cummins, J.R. Epps, V. Sethu, M. Breakspear, and R. Goecke Modeling spectral variability for the classification of depressed speech Proceedings of Interspeech 2013, ISCA 2013 857 861
12. N. Cummins, J.R. Epps, and E. Ambikairajah Spectro-temporal analysis of speech affected by depression and psychomotor retardation Proceedings of International Conference on Acoustics, Speech, and Signal Processing (ICASSP), IEEE 2013 7542 7546
13. J.R. Williamson, T.F. Quatieri, B.S. Helfer, R. Horwitz, B. Yu, and D.D. Mehta Vocal biomarkers of depression based on motor incoordination Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge, AVEC '13, ACM 2013 41 48 10.1145/2512530.2512531
14. A.C. Trevino, T.F. Quatieri, and N. Malyska Phonologically-based biomarkers for major depressive disorder EURASIP Journal on Advances in Signal Processing 42 2011
15. D. Sturim, P. Torres-Carrasquillo, T. Quatieri, N. Malyska, and A. McCree Automatic detection of depression in speech using gaussian mixture modeling with factor analysis Proceedings of Interspeech 2011 2011 2981 2984
16. G. Stratou, S. Scherer, J. Gratch, and L.-P. Morency Automatic nonverbal behavior indicators of depression and ptsd: exploring gender differences Proceedings of International Conference on Affective Computing and Intelligent Interaction (ACII), IEEE 2013
17. Y. Zhou, S. Scherer, D. Devault, J. Gratch, G. Stratou, L.-P. Morency, and J. Cassell Multimodal prediction of psychological disorder: learning nonverbal commonality in adjacency pairs Proceedings of Workshop Series on the Semantics and Pragmatics of Dialogue 2013
18. J. Gratch, L.-P. Morency, S. Scherer, G. Stratou, J. Boberg, S. Koenig, T. Adamson, and A. Rizzo User-state sensing for virtual health agents and telehealth applications Studies in Health Technology and Informatics 184 2012 151 157
19. S. Scherer, G. Stratou, J. Gratch, and L.-P. Morency Investigating voice quality as a speaker-independent indicator of depression and ptsd Proceedings of Interspeech 2013, ISCA 2013 847 851
20. S. Scherer, G. Stratou, and L.-P. Morency Audiovisual behavior descriptors for depression assessment Proceedings of International Conference on Multimdoal Interaction, ACM 2013
21. D. DeVault, K. Georgilia, R. Artstein, F. Morbini, D. Traum, S. Scherer, A. Rizzo, and L.-P. Morency Verbal indicators of psychological distress in interactive dialogue with a virtual human Proceedings of SigDial 2013 2013 Association for Computational Linguistics 193 202
22. D. DeVault, R. Artstein, and G. Benn et al. Simsensei: A virtual human interviewer for healthcare decision support in: Proceedings of Autonomous Agents and Multiagent Systems (AAMAS), to appear 2014 1061 1068
23. J. Gratch, R. Artstein, G. Lucas, S. Scherer, A. Nazarian, R. Wood, J. Boberg, D. DeVault, S. Marsella, D. Traum, A. Rizzo, and L.-P. Morency The distress analysis interview corpus of human and computer interviews Proceedings of Language Resources and Evaluation Conference (LREC) 2014
24. P. Waxer Nonverbal cues for depression J. Abnorm. Psychol. 83 3 1974 319 322
25. J.T.M. Schelde Major depression: behavioral markers of depression and recovery J. Nerv. Ment. Dis. 186 3 1998 133 140 (Pubitemid 28124327)
26. J.E. Perez, and R.E. Riggio Nonverbal social skills and psychopathology Nonverbal Behavior in Clinical Settings 2003 Oxford University Press 17 44
27. L.M. Bylsam, B.H. Morris, and J. Rottenberg A meta-analysis of emotional reactivity in major depressive disorder Clin. Psychol. Rev. 28 2008 676 691 (Pubitemid 351374605)
28. L.A. Fairbanks, M.T. McGuire, and C.J. Harris Nonverbal interaction of patients and therapists during psychiatric interviews J. Abnorm. Psychol. 91 2 1982 109 119
29. J.A. Hall, J.A. Harrigan, and R. Rosenthal Nonverbal behavior in clinician-patient interaction Appl. Prev. Psychol. 4 1 1995 21 37
30. A. Kirsch, and S. Brunnhuber Facial expression and experience of emotions in psychodynamic interviews with patients with ptsd in comparison to healthy subjects Psychopathology 40 5 2007 296 302 (Pubitemid 47254514)
31. R. Menke Examining nonverbal shame markers among post-pregnancy women with maltreatment histories Ph.D. thesis 2011 Wayne State University
32. P. Ekman, and W.V. Friesen The repertoire of nonverbal behavior: categories, origins, usage, and coding Semiotica 1 1969 49 98
33. A. Nilsonne Speech characteristics as indicators of depressive illness Acta Psychiatr. Scand. 77 3 1988 253 263
34. J.K. Darby, N. Simmons, and P.A. Berger Speech and voice parameters of depression: a pilot study J. Commun. Disord. 17 2 1984 75 85
35. A.J. Flint, S.E. Black, I. Campbell-Taylor, G.F.G. Gailey, and C. Levinton Abnormal speech articulation, psychomotor retardation, and subcortical dysfunction in major depression J. Psychiatr. Res. 27 3 1993 309 319 (Pubitemid 23344951)
36. M. Elliott, M.A. Clements, J.W. Peifer, and L. Weisser Critical analysis of the impact of glottal features in the classification of clinical depression in speech IEEE Trans. Biomed. Eng. 55 1 2008 96 107 (Pubitemid 350308415)
37. Y. Yang, C. Fairbairn, and J.F. Cohn Detecting depression severity from vocal prosody IEEE Trans. Affect. Comput. 4 2 2013 142 150
38. J.M. Girard, J.F. Cohn, M.H. Mahoor, S. Mavadati, and D.P. Rosenwald Social risk and depression: evidence from manual and automatic facial expression analysis Proceedings of IEEE Conference on Automatic Face and Gesture Recognition, IEEE 2013
39. J.A. Russell, and L.F. Barrett Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant J. Pers. Soc. Psychol. 76 5 1999 805 819
40. J.D. Elhai, L. de Francisco Carvalho, F.K. Miguel, P.A. Palmieri, R. Primi, and B. Christopher Frueh Testing whether posttraumatic stress disorder and major depressive disorder are similar or unique constructs J. Anxiety Disord. 25 3 2011 404 410
41. P.J. Bieling, M.M. Antony, and R.P. Swinson The state-trait anxiety inventory, trait version: structure and content re-examined Behav. Res. Ther. 36 7-8 1998 777 788 (Pubitemid 28339797)
42. G.N. Marshall, T.L. Schell, and J.N.V. Miles All ptsd symptoms are highly associated with general distress: ramifications for the dysphoria symptom cluster J. Abnorm. Psychol. 119 1 2010 126 135
43. P.A. Arbisi, M.E. Kaler, S.M. Kehle-Forbes, C.R. Erbes, M.A. Polusny, and P. Thuras The predictive validity of the ptsd checklist in a nonclinical sample of combat-exposed national guard troops Psychol. Assess. 4 24 2012 1034 1040
44. E.B. Blanchard, J. Jones-Alexander, T. Buckley, and C. Forneris Psychometric properties of the ptsd checklist (pcl) Behav. Res. Ther. 34 8 1996 669 673 (Pubitemid 26316912)
45. E.E. Bolton, M.J. Gray, and B.T. Litz A cross-lagged analysis of the relationship between symptoms of ptsd and retrospective reports of exposure J. Anxiety Disord. 20 7 2006 877 895 (Pubitemid 44247530)
46. C.W. Hoge, C.A. Castro, S.C. Messer, D. McGurk, D.I. Cotting, and R.L. Koffman Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care N. Engl. J. Med. 351 1 2004 13 22 (Pubitemid 38857243)
47. C.D. Spielberger, R.L. Gorsuch, and R.E. Lushene Manual for the State-Trait Anxiety Inventory 1970 Consulting Psychologists Press
48. A. Beck, N. Epstein, G. Brown, and R. Steer An inventory for measuring clinical anxiety: psychometric properties J. Consult. Clin. Psychol. 56 1988 893 897 (Pubitemid 19008360)
49. A. Sesti State trait anxiety inventory in medication clinical trials Qual. Life Newsl. 25 2000 15 16
50. K. Kroenke, and R.L. Spitzer The phq-9: a new depression and diagnostic severity measure Psychiatr. Ann. 32 2002 509 521
51. K. Kroenke, R.L. Spitzer, and J.B.W. Williams The phq-9 J. Gen. Intern. Med. 16 9 2001 606 613
52. D. Campbell, B. Felker, C.-F. Liu, E. Yano, J. Kirchner, D. Chan, L. Rubenstein, and E. Chaney Prevalence of depression-PTSD comorbidity: implications for clinical practice guidelines and primary care-based interventions J. Gen. Intern. Med. 22 2007 711 718 10.1007/s11606-006-0101-4 (Pubitemid 46763950)
53. J. Wagner, F. Lingenfelser, N. Bee, and E. André Social signal interpretation (ssi) KI - Kuenstliche Intelligenz 25 2011 251 256 10.1007/s13218-011-0115-x
54. G. Degottex, J. Kane, T. Drugman, T. Raitio, and S. Scherer Covarep - a collaborative voice analysis repository for speech technologies To Appear in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2014) 2014
55. S. Scherer, J.P. Pestian, and L.-P. Morency Investigating the speech characteristics of suicidal adolescents Proceedings of International Conference on Acoustics, Speech, and Signal Processing (ICASSP), IEEE 2013 709 713
56. S. Scherer, J. Kane, C. Gobl, and F. Schwenker Investigating fuzzy-input fuzzy-output support vector machines for robust voice quality classification Comput. Speech Lang. 27 1 2013 263 287 10.1016/j.csl.2012.06.001
57. J. Kane, S. Scherer, M. Aylett, L.-P. Morency, and C. Gobl Speaker and language independent voice quality classification applied to unlabelled corpora of expressive speech Proceedings of International Conference on Acoustics, Speech, and Signal Processing (ICASSP), IEEE 2013 7982 7986
58. P. Alku, T. Bäckström, and E. Vilkman Glottal wave analysis with pitch synchronous iterative adaptive inverse filtering Speech Comm. 11 2-3 1992 109 118 (Pubitemid 23572504)
59. P. Alku, T. Bäckström, and E. Vilkman Normalized amplitude quotient for parameterization of the glottal flow J. Acoust. Soc. Am. 112 2 2002 701 710 (Pubitemid 34855925)
60. N. Campbell, and P. Mokhtari Voice quality: the 4th prosodic dimension Proceedings of the 15th International Congress of Phonetic Sciences (ICPhS'03), ICPhS 2003 2417 2420
61. C. Gobl, and A.N. Chasaide Amplitude-based source parameters for measuring voice quality Proceedings of ISCA Tutorial and Research Workshop on Voice Quality: Functions, Analysis and Synthesis (VOQUAL'03), ISCA 2003 151 156
62. T. Hacki Klassifizierung von glottisdysfunktionen mit hilfe der elektroglottographie Folia Phoniatr. 1989 43 48 (Pubitemid 19174971)
63. H. Hanson, K. Stevens, H. Kuo, M. Chen, and J. Slifka Towards models of phonation J. Phon. 29 2001 451 480
64. N. Henrich, C. d'Alessandro, and B. Doval Spectral correlates of voice open quotient and glottal flow asymmetry: theory, limits and experimental data Proceedings of EUROSPEECH, Scandanavia 2001 47 50
65. R. Timcke, H. von Leden, and P. Moore Laryngeal vibrations: measurements of the glottic wave. Part 1: the normal vibratory cycle Arch. Otolaryngol. Head Neck Surg. 68 1 1958 1 19
66. D. Talkin A robust algorithm for pitch tracking W.B. Kleijn, K.K. Paliwal, Speech Coding and Synthesis 1995 Elsevier 495 517
67. F. Itakura Minimum prediction residual principle applied to speech recognition IEEE Trans. Acoust. Speech Signal Process. 23 1975 67 72
68. K. Krippendorff Agreement and information in the reliability of coding Communication Methods and Measures 5 (2) 2011 93 112
69. H. Lausberg, and H. Sloetjes Coding gestural behavior with the NEUROGES-ELAN system Behav. Res. Methods 41 3 2009 841 849 (URL http://www.lat-mpi.eu/tools/elan/)
70. B. Wildman, M. Erickson, and R. Kent The effect of two training procedures on observer agreement and variability of behavior ratings Child Dev. 1975 520 524
71. F. Harris, and B. Lahey Recording system bias in direct observational methodology: a review and critical analysis of factors causing inaccurate coding behavior Clin. Psychol. Rev. 2 4 1982 539 556
72. P. Ekman, and W. Friesen Facial Action Coding System: A Technique for the Measurement of Facial Movement 1978 Consulting Psychologists Press Palo Alto
73. A.L. Bouhuys, and R.H. van den Hoofdakker The interrelatedness of observed behavior of depressed patients and of a psychiatrist: an ethological study on mutual influence J. Affect. Disord. 23 1991 63 74
74. E.N. Geerts, A.L. Bouhuys, and R.H. van den Hoofdakker Nonverbal attunement between depressed patients and an interviewer predicts subsequent improvement J. Affect. Disord. 40 1-2 1999 15 21 (Pubitemid 26294846)