Toward automating a human behavioral coding system for married couples' interactions using speech acoustic features

Speech Communication

Volumn 55, Issue 1, 2013, Pages 1-21

  Black, Matthew P ^a   Katsamanis, Athanasios ^a   Baucom, Brian R ^a   Lee, Chi Chun ^a   Lammert, Adam C ^a   Christensen, Andrew ^b   Georgiou, Panayiotis G ^a   Narayanan, Shrikanth S ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Behavioral signal processing (BSP); Couple therapy; Dyadic interaction; Emotion recognition; Human behavior analysis; Prosody

Indexed keywords

BEHAVIORAL SIGNAL PROCESSING (BSP); COUPLE THERAPY; DYADIC INTERACTION; EMOTION RECOGNITION; HUMAN BEHAVIORS; PROSODY;

ACOUSTIC PROPERTIES; SIGNAL ENCODING; SOCIAL SCIENCES; VIDEO RECORDING;

BEHAVIORAL RESEARCH;

EID: 84870290069     PISSN: 01676393     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.specom.2011.12.003     Document Type: Article

References (79)

1. Atkins, D.; Milbright, S.A.; Dueck, A.; Reimer, K.; Christensen, A.; 2005. The language of therapy: The promises and hurdles of computational linguistics. In: Annual Meeting of the Association for Behavioral and Cognitive Therapies, Washington, D.C.
2. Batliner, A.; Schuller, B.; Seppi, D.; Steidl, S.; Devillers, L.; Vidrascu, L.; Vogt, T.; Aharonson, V.; Amir, N.; 2011. The Automatic Recognition of Emotions in Speech. Emotion-Oriented Systems: The Humaine Handbook Cognitive Technologies, pp. 71-99.
3. D.H. Baucom, V. Shoham, K.T. Mueser, A.D. Daiuto, and T.R. Stickle Empirically supported couple and family interventions for marital distress and adult mental health problems J. Consult. Clin. Psychol. 66 1998 53 88 (Pubitemid 28086810)
4. B.R. Baucom, K. Eldridge, J. Jones, M. Sevier, M. Clements, H. Markman, S. Stanley, S.L. Sayers, T. Sher, and A. Christensen Relative contributions of relationship distress and depression to communication patterns in couples J. Soc. Clin. Psychol. 26 2007 689 707
5. B.R. Baucom, D.C. Atkins, L.E. Simpson, and A. Christensen Prediction of response to treatment in a randomized clinical trial of couple therapy: A 2-year follow-up J. Consult. Clin. Psychol. 77 2009 160 173
6. J.G. Beck, J. Davila, S. Farrow, and D. Grant When the heat is on: Romantic partner responses influence distress in socially anxious women Behav. Res. Ther. 44 2006 737 748
7. Black, M.P.; Katsamanis, A.; Lee, C.C.; Lammert, A.C.; Baucom, B.R.; Christensen, A.; Georgiou, P.G.; Narayanan, S.S. 2010. Automatic classification of married couples' behavior using audio features. In: Proc. Interspeech, Makuhari, Japan. pp. 2030-2033.
8. Black, M.P.; Georgiou, P.G.; Katsamanis, A.; Baucom, B.R.; Narayanan, S.S. 2011. "You made me do it": Classification of blame in married couples' interactions by fusing automatically derived speech and language information, in: Proc. Interspeech, Florence, Italy.
9. P.P.G. Boersma Praat, a system for doing phonetics by computer Glot Internat. 5 2001 341 345
10. M. Brüne, C. Sonntag, M. Abdel-Hamid, C. Lehmkämper, G. Juckel, and A. Troisi Nonverbal behavior during standardized interviews in patients with schizophrenia spectrum disorders J. Nerv. Ment. Dis. 196 2008 282 288 (Pubitemid 351548728)
11. M. Bulut, and S.S. Narayanan On the robustness of overall F0-only modifications to the perception of emotions in speech J. Acoust. Soc. Amer 123 2008 4547 4558 (Pubitemid 351810187)
12. Burkhardt, F.; Polzehl, T.; Stegmann, J.; Metze, F.; Huber, R. 2009. Detecting real life anger, in: Proc. IEEE Int'l Conf. Acous.; Speech, and Signal Processing, Taipei, Taiwan. pp. 4761-4764.
13. Busso, C.; Bulut, M.; Lee, S.; Narayanan, S.S. 2009a. Fundamental frequency analysis for speech emotion processing. In: Hancil, S. (Ed.), The Role of Prosody in Affective Speech. Peter Lang Publishing Group, Berlin, Germany, pp. 309-337.
14. C. Busso, S. Lee, and S.S. Narayanan Analysis of emotionally salient aspects of fundamental frequency for emotion detection IEEE Trans. Audio, Speech, Lang. Process. 17 2009 582 596
15. Campbell, N. 2000. Databases of emotional speech, in: ISCA Tutorial and Research Workshop (ITRW) on Speech and Emotion.
16. Chen, K.; Hasegawa-Johnson, M.; Cohen, A. 2004. An automatic prosody labeling system using ANN-based syntactic-prosodic model and GMM-based acoustic-prosodic model, in: Proc. IEEE Int'l Conf. Acous.; Speech, and Signal Processing, Montreal, Quebec, Canada. pp. 509-512.
17. A. Christensen, and C.L. Heavey Gender differences in marital conflict: The demand/withdraw interaction pattern Gender Issues Contemp. Soc. 6 1990 113 141
18. A. Christensen, N.S. Jacobson, and J.C. Babcock Integrative behavioral couple therapy N.S. Jacobsen, A.S. Gurman, Clinical Handbook of Marital Therapy second ed. 1995 Guilford Press New York 31 64
19. A. Christensen, D.C. Atkins, S. Berns, J. Wheeler, D.H. Baucom, and L.E. Simpson Traditional versus integrative behavioral couple therapy for significantly and chronically distressed married couples J. Consult. Clin. Psychol. 72 2004 176 191 (Pubitemid 38393085)
20. A. Christensen, D.C. Atkins, J. Yi, D.H. Baucom, and W.H. George Couple and individual adjustment for 2 years following a randomized clinical trial comparing traditional versus integrative behavioral couple therapy J. Consult. Clin. Psychol. 74 2006 1180 1191 (Pubitemid 44950841)
21. A. Christensen, D.C. Atkins, B.R. Baucom, and J. Yi Marital status and satisfaction five years following a randomized clinical trial comparing traditional versus integrative behavioral couple therapy J. Consult. Clin. Psychol. 78 2010 225 235
22. R. Cowie Perceiving emotion: Towards a realistic understanding of the task Philos. Trans. Roy. Soc. B: Biological Sci. 364 2009 3515 3525
23. A. Coy, and J. Barker An automatic speech recognition system based on the scene analysis account of auditory perception Speech Comm. 49 2007 384 401 (Pubitemid 46670362)
24. A. de Cheveigné, and H. Kawahara YIN, a fundamental frequency estimator for speech and music J. Acoust. Soc. Amer 111 2002 1917 1930 (Pubitemid 34297247)
25. L. Devillers, and N. Campbell Special issue of computer speech and language on "affective speech in real-life interactions" Comput. Speech Lang. 25 2011 1 3
26. L. Devillers, L. Vidrascu, and L. Lamel Challenges in real-life emotion annotation and machine learning based detection Neural Networks 18 2005 407 422 (Pubitemid 40922648)
27. E. Douglas-Cowie, N. Campbell, R. Cowie, and P. Roach Emotional speech: Towards a new generation of databases Speech Comm. 40 2003 33 60
28. Douglas-Cowie, E.; Cowie, R.; Sneddon, I.; Cox, C.; Lowry, O.; McRorie, M.; Martin, J.C.; Devillers, L.; Abrilian, S.; Batliner, A.; Amir, N.; Karpouzis, K. 2007. The HUMAINE Database: Addressing the collection and annotation of naturalistic and induced emotional data, in: Affective Computing and Intelligent Interaction, Lisbon, Portugal. pp. 488-500.
29. Eyben, F.; Wöllmer, M.; Schuller, B. 2010. OpenSMILE - The Munich versatile and fast open-source audio feature extractor, in: ACM Multimedia, Firenze, Italy. pp. 1459-1462.
30. R. Fan, K. Chang, C. Hsieh, X. Wang, and C. Lin LIBLINEAR: A library for large linear classification J. Machine Learn. Res. 9 2008 1871 1874
31. S. Fredman, D. Baucom, D. Miklowitz, and S. Stanton Observed emotional involvement and overinvolvement in families of patients with bipolar disorder J. Fam. Psychol. 22 2008 71 79
32. Georgiou, P.G.; Black, M.P.; Lammert, A.C.; Baucom, B.R.; Narayanan, S.S. 2011. "That's aggravating, very aggravating": Is it possible to classify behaviors in couple interactions using automatically derived lexical features?, in: Affective Computing and Intelligent Interaction, Memphis, TN, USA.
33. P.K. Ghosh, A. Tsiartas, and S.S. Narayanan Robust voice activity detection using long-term signal variability IEEE Trans. Audio, Speech, Lang. Process. 19 2010 600 613
34. Gibson, J.; Katsamanis, A.; Black, M.P.; Narayanan, S.S. 2011. Automatic identification of salient acoustic instances in couples' behavioral interactions using Diverse Density Support Vector Machines, in: Proc. Interspeech, Florence, Italy.
35. C. Gobl, and A.N. Chasaide The role of voice quality in communicating emotion, mood and attitude Speech Comm. 40 2003 189 212
36. G.C. Gonzaga, B. Campos, and T. Bradbury Similarity, convergence, and relationship satisfaction in dating and married couples J.Personal. Soc. Psychol. 93 2007 34 48
37. J.M. Gottman, and L.J. Krokoff Marital interaction and satisfaction: A longitudinal view J. Consult. Clin. Psychol. 57 1989 47 52 (Pubitemid 19151645)
38. J.M. Gottman, H. Markman, and C. Notarius The topography of marital conflict: A sequential analysis of verbal and nonverbal behavior J. Marriage Fam. 39 1977 461 477 (Pubitemid 8166174)
39. M. Grimm, K. Kroschel, E. Mower, and S.S. Narayanan Primitives-based evaluation and estimation of emotions in speech Speech Comm. 49 2007 787 800 (Pubitemid 47268568)
40. K.J. Han, S. Kim, and S.S. Narayanan Strategies to improve the robustness of agglomerative hierarchical clustering under data source variation for speaker diarization IEEE Trans. Audio, Speech, Lang. Process. 16 2008 1590 1601
41. Heavey, C.; Gill, D.; Christensen, A. 2002. Couples interaction rating system 2 (CIRS2). University of California, Los Angeles. Los Angeles, CA, USA.
42. R. Heyman Observation of couple conflicts: Clinical assessment applications, stubborn truths, and shaky foundations Psychol. Assess. 13 2001 5 35 (Pubitemid 32233812)
43. Hops, H.; Wills, T.A.; Patterson, G.R.; Weiss, R.L. 1971. Marital Interaction Coding System. Technical Report. University of Oregon. Eugene, Oregon, USA.
44. Joachims, T. 1998. Text categorization with Support Vector Machines: Learning with many relevant features, in: European Conference on Machine Learning, Chemnitz, Germany. pp. 137-142. (Pubitemid 128067178)
45. Jones, J.; Christensen, A. 1998. Couples interaction study: Social support interaction rating system. University of California, Los Angeles. Los Angeles, CA, USA.
46. Jurafsky, D.; Ranganath, R.; McFarland, D. 2009. Extracting social meaning: Identifying interactional style in spoken conversation. In: Human Language Technologies, Boulder, CO, USA. pp. 638-646.
47. P.N. Juslin, and K.R. Scherer Vocal Expression of Affect The New Handbook of Methods in Nonverbal Behavior Research 2005 Oxford University Press Oxford, UK chapter 3, pp. 65-135
48. B.R. Karney, and T.N. Bradbury The longitudinal course of marital quality and stability: A review of theory, methods, and research Psychol. Bull. 118 1995 3 34
49. Katsamanis, A.; Black, M.P.; Georgiou, P.G.; Goldstein, L.; Narayanan, S.S. 2011a. SailAlign: Robust long speech-text alignment. In: Very-Large-Scale Phonetics Workshop, Philadelphia, PA, USA.
50. Katsamanis, A.; Gibson, J.; Black, M.P.; Narayanan, S.S. 2011b. Multiple instance learning for classification of human behavior observations. In: Affective Computing and Intelligent Interaction, Memphis, TN, USA.
51. D. Keen The use of non-verbal repair strategies by children with autism Res. Dev. Disabil. 26 2005 243 254 (Pubitemid 40164206)
52. P.K. Kerig, and D.H. Baucom Couple Observational Coding Systems 2004 Lawrence Erlbaum Mahwah, NJ, USA
53. C.M. Lee, and S.S. Narayanan Toward detecting emotions in spoken dialogs IEEE Trans. Speech Audio Process. 13 2005 293 303 (Pubitemid 40320247)
54. Lee, C.C.; Mower, E.; Busso, C.; Lee, S.; Narayanan, S.S. 2009. Emotion recognition using a hierarchical binary decision tree approach, in: Proc. Interspeech, Brighton, UK. pp. 320-323.
55. Lee, C.C.; Black, M.P.; Katsamanis, A.; Lammert, A.C.; Baucom, B.R.; Christensen, A.; Georgiou, P.G.; Narayanan, S.S.; 2010. Quantification of prosodic entrainment in affective spontaneous spoken interactions of married couples. In: Proc. Interspeech, Makuhari, Japan. pp. 793-796.
56. Lee, C.C.; Katsamanis, A.; Black, M.P.; Baucom, B.R.; Georgiou, P.G.; Narayanan, S.S. 2011. An analysis of PCA-based vocal entrainment measures in married couples' affective spoken interactions. In: Proc. Interspeech, Florence, Italy.
57. G. Margolin, P.H. Oliver, E.B. Gordis, H.G. O'Hearn, A.M. Medina, C.M. Ghosh, and L. Morland The nuts and bolts of behavioral observation of marital and family interaction Clin. Child Fam. Psychol. Rev. 1 1998 195 213 (Pubitemid 128691603)
58. G. Margolin, E.B. Gordis, and P.H. Oliver Links between marital and parent-child interactions: Moderating role of husband-to-wife aggression Dev. Psychopathol. 16 2004 753 771 (Pubitemid 351479992)
59. Q. McNemar Note on the sampling error of the difference between correlated proportions or percentages Psychometrika 12 1947 153 157
60. Mendenhall, W.; Sincich, T. 2007. Statistics for Engineering and the Sciences. Pearson Prentice Hall. chapter 7.8: Estimation of the Difference Between Two Population Proportions. pp. 302-303.
61. Moreno, P.J.; Joerg, C.; van Thong, J.M.; Glickman, O. 1998. A recursive algorithm for the forced alignment of very long audio segments. In: Proc. ICSLP, Sydney, Australia.
62. Murray, K. 2001. A study of automatic pitch tracker doubling/halving "errors". In: SIGdial Workshop on Discourse and Dialogue, Aalborg, Denmark.
63. M. O'Brien, R.S. John, G. Margolin, and O. Erel Reliability and diagnostic efficacy of parents' reports regarding children's exposure to marital aggression Violence and Victims 9 1994 45 62 (Pubitemid 24319350)
64. Ranganath, R.; Jurafsky, D.; McFarland, D. 2009. It's not you, it's me: Detecting flirting and its misperception in speed-dates. In: Conference on Empirical Methods in Natural Language Processing, Suntec City, Singapore. pp. 334-342.
65. Rozgić, V.; Xiao, B.; Katsamanis, A.; Baucom, B.; Georgiou, P.G.; Narayanan, S.S.; 2010. A new multichannel multimodal dyadic interaction database. In: Proc. Interspeech, Makuhari, Japan. pp. 1982-1985.
66. Rozgić, V.; Xiao, B.; Katsamanis, A.; Baucom, B.; Georgiou, P.G.; Narayanan, S.S. 2011. Estimation of ordinal approach-avoidance labels in dyadic interactions: Ordinal logistic regression approach. In: Proc. IEEE Int'l Conf. Acous.; Speech, and Signal Processing, Prague, Czech Republic. pp. 2368-2371.
67. Schuller, B.; Batliner, A.; Seppi, D.; Steidl, S.; Vogt, T.; Wagner, J.; Devillers, L.; Vidrascu, L.; Amir, N.; Kessous, L. 2007. The relevance of feature type for automatic classification of emotional user states: Low level descriptors and functionals. In: Proc. Interspeech, Antwerp, Belgium. pp. 2253-2256.
68. Schuller, B.; Wimmer, M.; Mösenlechner, L.; Kern, C.; Arsic, D.; Rigoll, G. 2008. Brute-forcing hierarchical functionals for paralinguistics: A waste of feature space?. In: Proc. IEEE Int'l Conf. Acous.; Speech, and Signal Processing, Las Vegas, NV, USA. pp. 4501-4504.
69. Schuller, B.; Steidl, S.; Batliner, A. 2009a. The Interspeech 2009 emotion challenge. In: Proc. Interspeech, Brighton, UK. pp. 312-315.
70. B. Schuller, M. Wöllmer, F. Eyben, and G. Rigoll Prosodic, spectral or voice quality? Feature type relevance for the discrimination of emotion pairs S. Hancil, The Role of Prosody in Affective Speech 2009 Peter Lang Publishing Group Berlin, Germany 285 307
71. Schuller, B.; Steidl, S.; Batliner, A.; Burkhardt, F.; Devillers, L.; Müller, C.; Narayanan, S.S. 2010. The Interspeech 2010 paralinguistic challenge. In: Proc. Interspeech, Makuhari, Japan. pp. 2794-2797.
72. Sevier, M.; Simpson, L.E.; Christensen, A. 2004. Demand/withdraw interaction coding. Lawrence Erlbaum, Mahwah, NJ, USA. Couple observational coding systems, pp. 159-172.
73. M. Sevier, K. Eldridge, J. Jones, B.D. Doss, and A. Christensen Observed communication and associations with satisfaction during traditional and integrative behavioral couple therapy Behavior Ther. 39 2008 137 150 (Pubitemid 351697898)
74. V. Shoham, M. Rohrbaugh, T. Stickle, and T. Jacob Demand-withdraw couple interaction moderates retention in cognitive-behavioral versus family-systems treatments for alcoholism J. Fam. Psychol. 12 1998 557 577 (Pubitemid 128367118)
75. S.E. Tranter, and D.A. Reynolds An overview of automatic speaker diarization systems IEEE Trans. Audio, Speech, Lang. Process. 14 2006 1557 1565 (Pubitemid 46547580)
76. Traunmüller, H.; Eriksson, A. 1994. The frequency range of the voice fundamental in the speech of male and female adults. Technical Report. Linguistics Department, University of Stockholm. Stockholm, Sweden.
77. A. Vinciarelli, M. Pantic, and H. Bourlard Social signal processing: Survey of an emerging domain Image Vision Comput. 27 2009 1743 1759
78. K. Williams-Baucom, D. Atkins, M. Sevier, K. Eldridge, and A. Christensen "You" and "I" need to talk about "us": Linguistic patterns in marital interactions Pers. Relationship. 17 2010 41 56
79. S. Yildirim, S.S. Narayanan, and A. Potamianos Detecting emotional state of a child in a conversational computer game Comput. Speech Lang. 25 2010 29 44