Combining long short-term memory and dynamic bayesian networks for incremental emotion-sensitive artificial listening

IEEE Journal on Selected Topics in Signal Processing

Volumn 4, Issue 5, 2010, Pages 867-881

  Wöllmer, Martin ^a   Schuller, Björn ^a   Eyben, Florian ^a   Rigoll, Gerhard ^a  

^a TECHNICAL UNIVERSITY OF MUNICH   (Germany)

Author keywords

Dynamic Bayesian networks (DBNs); emotion recognition; intelligent environments; long short term memory (LSTM); recurrent neural nets; virtual agents

Indexed keywords

DYNAMIC BAYESIAN NETWORKS; EMOTION RECOGNITION; INTELLIGENT ENVIRONMENT; NEURAL NET; SHORT TERM MEMORY; VIRTUAL AGENT;

BRAIN; DISTRIBUTED PARAMETER NETWORKS; FEATURE EXTRACTION; FORECASTING; INFERENCE ENGINES; INTELLIGENT AGENTS; INTELLIGENT NETWORKS; INTELLIGENT VIRTUAL AGENTS; LINGUISTICS; RECURRENT NEURAL NETWORKS; SPEECH RECOGNITION; VIRTUAL REALITY;

BAYESIAN NETWORKS;

EID: 77956721304     PISSN: 19324553     EISSN: None     Source Type: Journal    
DOI: 10.1109/JSTSP.2010.2057200     Document Type: Article

References (90)

1. M. T. Vo and A. Waibel, "Multimodal human-computer interaction," in Proc. ISSD'93, Waseda, Japan, 1993.
2. S. Furui, "Toward the ultimate synthesis/recognition," Proc. Nat. Acad. Sci. USA, vol.92, no.22, pp. 10 040-10 045, 1995.
3. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. G. Taylor, "Emotion recognition in human-computer interaction," IEEE Signal Process. Mag., vol.18, no.1, pp. 32-80, Feb. 2001.
4. R. W. Picard, "Toward agents that recognize emotion," in Actes Proc. IMAGINA, 1998, pp. 153-165.
5. E. Shriberg, "How peoply really talk and why engineers should care," in Proc. Interspeech, Lisbon, Portugal, 2005, pp. 1781-1784.
6. Z. Zeng, M. Pantic, G. I. Rosiman, and T. S. Huang, "A survey of affect recognition methods: Audio, visual, and spontaneous expressions," Trans. Pattern Anal. Mach. Intell., vol.31, no.1, pp. 39-58, Jan. 2009.
7. R. Picard, Affective Computing. Cambridge, MA: MIT Press, 1997.
8. S. Casale, A. Russo, G. Scebba, and S. Serrano, "Speech emotion classification using machine learning algorithms," in Proc. IEEE Int. Conf. Semantic Comput., 2008, pp. 158-165.
9. B. Schuller, M. Wimmer, L. Mösenlechner, C. Kern, D. Arsic, and G. Rigoll, "Brute-forcing hierarchical functionals for paralinguistics: A waste of feature space?," in Proc. ICASSP'08, Las Vegas, NV, 2008, pp. 4501-4504.
10. M. Lee and S. S. Narayanan, "Toward detecting emotions in spoken dialogs," IEEE Trans. Speech Audio Process., vol.13, no.2, pp. 293-303, Mar. 2005.
11. L. Devillers, L. Vidrascu, and L. Lamel, "Challenges in real-life emotion annotation and machine learning based detection," Neural Netw., vol.18, no.4, pp. 407-422, 2005.
12. V. Petrushin, "Emotion in speech: Recognition and application to call centers," Artif. Neural Netw Eng. (ANNIE), pp. 7-12, 1999.
13. B. Schuller, R. Müller, B. Hörnler, A. Hoethker, H. Konosu, and G. Rigoll, "Audiovisual recognition of spontaneous interest within conversations," in Proc. Int. Conf. Multimodal Interfaces, ACM SIGHI, Nagoya, Japan, 2007, pp. 30-37.
14. B. Schuller, R. Müller, F. Eyben, J. Gast, B. Hörnler, M. Wöllmer, G. Rigoll, A. Höthker, and H. Konosu, "Being bored recognising natural interest by extensive audiovisual integration for real-life application," Image Vis. Comput. J.(IMAVIS), Special Iss. Vis. Multimodal Anal. Human Spontaneous Behavior, vol.27, no.12, pp. 1760-1774, 2009.
15. A. Batliner, S. Steidl, and E. Nöth, "Releasing a thoroughly annotated and processed spontaneous emotional database: The FAU Aibo Emotion Corpus," in Proc. Satellite Workshop of LREC 2008 Corpora Res. Emotion and Affect, L. Devillers, J. C. Martin, R. Cowie, E. Douglas-Cowie, and A. Batliner, Eds., 2008, pp. 28-31.
16. S. Steininger, F. Schiel, O. Dioubina, and S. Raubold, "Development of user-state conventions for the multimodal corpus in smartkom," in Proc. Workshop Multimodal Resources Multimodal Syst. Eval., Las Palmas, Spain, 2002, pp. 33-37.
17. B. Schuller, G. Rigoll, S. Can, and H. Feussner, "Emotion sensitive speech control for human-robot interaction in minimal invasive surgery," in Proc. 17th Int. Symp. Robot Human Interactive Commun. RO-MAN'08, Munich, Germany, 2008, pp. 453-458.
18. J. Hansen and S. Bou-Ghazale, "Getting started with SUSAS: A speech under simulated and actual stress database," in Proc. Eurospeech, Rhodes, Greece, 1997, pp. 1743-1746.
19. T. Vogt and E. André, "Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition," in Proc. ICME, Amsterdam, The Netherlands, 2005.
20. F. Burkhardt, A. Paeschke, M. Rolfes, W. Sendlmeier, and B. Weiss, "A database of German emotional speech," in Proc. Interspeech'05, Lisbon, Portugal, 2005, pp. 1517-1520.
21. I. S. Engberg, A. V. Hansen, O. Andersen, and P. Dalsgaard, "Design, recording and verification of a Danish emotional speech database," in Proc. Eurospeech, Rhodes, Greece, 1997, pp. 1695-1698.
22. O. Martin, I. Kotsia, B. Macq, and I. Pitas, "The eNTERFACE'05 Audiovisual Emotion Database," in Proc. IEEE Workshop Multimedia Database Management, Atlanta, Georgia, 2006.
23. B. Schuller, S. Steidl, and A. Batliner, "The Interspeech 2009 emotion challenge," in Proc. Interspeech, Brighton, U.K., 2009, pp. 312-315.
24. S. Steidl, B. Schuller, A. Batliner, and D. Seppi, "The hinterland of emotions: Facing the open-microphone challenge," in Proc. ACII, Amsterdam, The Netherlands, 2009, pp. 690-697.
25. M. Schröder, R. Cowie, D. Heylen, M. Pantic, C. Pelachaud, and B. Schuller, "Towards responsive sensitive artificial listeners," in Proc. 4th Int. Workshop Human-Comput. Convers., Bellagio, Italy, 2008.
26. M. Wöllmer, F. Eyben, S. Reiter, B. Schuller, C. Cox, E. Douglas-Cowie, and R. Cowie, "Abandoning emotion classes-Towards continuous emotion recognition with modelling of long-range dependencies," in Proc. Interspeech, Brisbane, Australia, 2008, pp. 597-600.
27. S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol.9, no.8, pp. 1735-1780, 1997.
28. A. Graves and J. Schmidhuber, "Framewise phoneme classification with bidirectional LSTM and other neural network architectures," Neural Netw., vol.18, no.5-6, pp. 602-610, Jun. 2005.
29. M. Wöllmer, F. Eyben, J. Keshet, A. Graves, B. Schuller, and G. Rigoll, "Robust discriminative keyword spotting for emotionally colored spontaneous speech using bidirectional LSTM networks," in Proc. ICASSP, Taipei, Taiwan, 2009, pp. 3949-3952.
30. M. Wöllmer, F. Eyben, B. Schuller, Y. Sun, T. Moosmayr, and N. Nguyen-Thien, "Robust in-car spelling recognition-A tandem BLSTM-HMM approach," in Proc. Interspeech, Brighton, U.K., 2009, pp. 2507-2510.
31. J. Nicholson, K. Takahashi, and R. Nakatsu, "Emotion recognition in speech using neural networks," Neural Comput. Applicat., vol.9, pp. 290-296, 2000.
32. Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, "A survey of affect recognition methods: Audio, visual, and spontaneous expressions," IEEE Trans. Pattern Anal. Mach. Intell., vol.31, no.1, pp. 39-58, Jan. 2009.
33. B. Schuller, S. Reiter, and G. Rigoll, "Evolutionary feature generation in speech emotion recognition," in Proc. ICME'06, Toronto, ON, Canada, 2006, pp. 5-8.
34. M. Streit, A. Batliner, and T. Portele, "Emotions analysis and emotionhandling subdialogues," in SmartKom: Foundations of Multimodal Dialogue Systems, W. Wahlster, Ed. Berlin, Germany: Springer, 2006, pp. 317-332.
35. S. Steidl, in Automatic Classification of Emotion-Related User States in Spontaneous Children's Speech, Berlin, Germany, 2009, Logos Verlag.
36. B. Schuller and G. Rigoll, "Timing levels in segment-based speech emotion recognition," in Proc. Interspeech'06, Pittsburgh, PA, 2006, pp. 1818-1821, ISCA.
37. B. Schuller, B. Vlasenko, R. Minguez, G. Rigoll, and A. Wendemuth, "Comparing one and two-stage acoustic modeling in the recognition of emotion in speech," in Proc. ASRU'07, Kyoto, Japan, 2007, pp. 596-600.
38. B. Schuller, G. Rigoll, and M. Lang, "Hidden Markov model-based speech emotion recognition," in Proc. ICASSP'03, Hong Kong, China, 2003, pp. 1-4.
39. B. Vlasenko, B. Schuller, A. Wendemuth, and G. Rigoll, "Frame vs. turnlevel: Emotion recognition from speech considering static and dynamic processing," in Proc. ACII'07, Lisbon, Portugal, A. Paiva, Ed., Heidelberg, Germany, 2007, vol.LNCS 4738, pp. 139-147, Springer Berlin.
40. M. Schuster and K. K. Paliwal, "Bidirectional recurrent neural networks," IEEE Trans. Signal Process., vol.45, no.11, pp. 2673-2681, Nov. 1997.
41. M. Wöllmer, F. Eyben, A. Graves, B. Schuller, and G. Rigoll, "A tandem BLSTM-DBN architecture for keyword spotting with enhanced context modeling," in Proc. NOLISP, Vic, Spain, 2009.
42. E. Douglas-Cowie, R. Cowie, I. Sneddon, C. Cox, O. Lowry, M. McRorie, J. C. Martin, L. Devillers, S. Abrilian, A. Batliner, N. Amir, and K. Karpouzis, "The HUMAINE database: Addressing the collection and annotation of naturalistic and induced emotional data," in Affective Computing and Intelligent Interaction. Berlin/Heidelberg, Germany: Springer, 2007, vol.4738/2007, pp. 488-500 [Online]. Available: http://emotionresearch.net/ download/pilot-db/
43. C. Lee, C. Busso, S. Lee, and S. Narayanan, "Modeling mutual influence of interlocutor emotion states in dyadic spoken interactions," in Proc. Interspeech, Brighton, U.K., 2009, pp. 1983-1986.
44. R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, and M. Schröder, "FEELtrace: An instrument for recording perceived emotion in real time," in Proc. ISCA Workshop Speech Emotion, 2000, pp. 19-24.
45. M. Grimm, K. Kroschel, and S. Narayanan, "The Vera am Mittag German audio-visual emotional speech database," in Proc. IEEE Int. Conf. Multimedia and Expo (ICME), Hannover, Germany, 2008, pp. 865-868.
46. D. Seppi, A. Batliner, B. Schuller, S. Steidl, T. Vogt, J.Wagner, L. Devillers, L. Vidrascu, N. Amir, and V. Aharonson, "Patterns, prototypes, performance: Classifying emotional user states," in Proc. Interspeech, Brisbane, Australia, 2008, pp. 601-604.
47. F. Eyben, M. Wöllmer, and B. Schuller, "openEAR-Introducing the Munich open-source emotion and affect recognition toolkit," in Proc. ACII, Amsterdam, The Netherlands, 2009, pp. 576-581.
48. M. Schröder, L. Devillers, K. Karpouzis, J.-C. Martin, C. Pelachaud, C. Peter, H. Pirker, B. Schuller, J. Tao, and I. Wilson, "What should a generic emotion markup language be able to represent?," in Affective Computing and Intelligent Interaction,A. Paiva,R. Prada, and R. W. Picard, Eds. Berlin-Heidelberg, Germany: Springer, 2007, pp. 440-451.
49. P. Baggia, F. Burkhardt, C. Pelachaud, C. Peter, B. Schuller, I.Wilson, and E. Zovato, "Elements of an EmotionML 1.0," in W3C Incubator Group Report, M. Schröder, Ed., 2008, W3C.
50. B. Schuller and G. Rigoll, "Recognising interest in conversational speech-comparing bag of frames and supra-segmental features," in Proc. Interspeech, Brighton, U.K., 2009, pp. 1999-2002.
51. A. Quattoni, S. Wang, L. P. Morency, M. Collins, and T. Darrell, "Hidden conditional random fields," IEEE Trans. Pattern Anal. Mach. Intell., vol.29, no.10, pp. 1848-1853, Oct. 2007.
52. S. Hochreiter, Y. Bengio, P. Frasconi, and J. Schmidhuber, "Gradient flow in recurrent nets: The difficulty of learning long-term dependencies," in A Field Guide to Dynamical Recurrent Neural Networks, S. C. Kremer and J. F. Kolen, Eds. Piscataway, NJ: IEEE Press, 2001.
53. A. Graves, S. Fernandez, and J. Schmidhuber, "Bidirectional LSTM networks for improved phoneme classification and recognition," in Proc. ICANN, Warsaw, Poland, 2005, vol.18, pp. 602-610. (Pubitemid 43186580)
54. S. Fernandez, A. Graves, and J. Schmidhuber, "Phoneme Recognition in TIMIT With BLSTM-CTC," IDSIA, Tech. Rep., 2008.
55. M. A. Hall, "Correlation-based feature selection for machine learning," Ph.D. dissertation, Univ. of Waikato, Hamilton, New Zealand, 1999.
56. I. H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques, 2nd ed. San Francisco, CA: Morgan Kaufmann, 2005.
57. S. Arunachalam, D. Gould, E. Anderson, D. Byrd, and S. Narayanan, "Politeness and frustration language in child-machine interactions," in Proc. Eurospeech, Aalborg, Denmark, 2001, pp. 2675-2678.
58. Z. J. Chuang and C. H. Wu, "Emotion recognition using acoustic features and textual content," in Proc. ICME, 2004, pp. 53-56.
59. K. Dupuis and K. Pichora-Fuller, "Use of lexical and affective prosodic cues to emotion by younger and older adults," in Proc. Interspeech, Antwerp, Belgium, 2007, pp. 2237-2240.
60. C. Elliott, "The affective reasoner: A process model of emotions in a multi-agent system," Ph.D. dissertation, Northwestern Univ., Evanston, IL, 1992.
61. R. Cowie, E. Douglas-Cowie, B. Apolloni, J. Taylor, A. Romano, and W. Fellenz, "What a neural net needs to know about emotion words," Comput. Intell. Applicat., pp. 109-114, 1999, N. Mastorakis, Ed..
62. D. Litman and K. Forbes, "Recognizing emotions from student speech in tutoring dialogues," in Proc. ASRU, Virgin Islands, 2003, pp. 25-30.
63. X. Zhe and A. Boucouvalas, "Text-to-emotion engine for real time internet communication," in Proc. Int. Symp. Commun. Syst., Netw., DSPs, 2002, pp. 164-168, Staffordshire Univ., Stoke-on-Trent, U.K.
64. B. Goertzel, K. Silverman, C. Hartley, S. Bugaj, and M. Ross, "The baby webmind project," in Proc. Annu. Conf. Soc. Study Artif. Intell. Simul. Behav. (AISB), 2000.
65. T. Wu, F. Khan, T. Fisher, L. Shuler, and W. Pottenger, "Posting act tagging using transformation-based learning," in Foundations of Data Mining and Knowledge Discovery, T. Y. Lin, S. Ohsuga, C. J. Liau, X. Hu, and S. Tsumoto, Eds., 2005, pp. 319-331.
66. H. Liu, H. Liebermann, and T. Selker, "A model of textual affect sensing using real-world knowledge," in Proc. IUI, 2003, pp. 125-132.
67. B. Schuller,G. Rigoll, and M. Lang, "Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture," in Proc. ICASSP, 2004, pp. 577-580.
68. J. Breese and G. Ball, "Modeling emotional state and personality for conversational agents," Microsoft, Tech. Rep., 1998.
69. G. Rigoll, R. Müller, and B. Schuller, "Speech emotion recognition exploiting acoustic and linguistic information sources," in Proc. SPECOM, Patras, Greece, 2005, pp. 61-67.
70. T. S. Polzin and A. Waibel, "Emotion-sensitive human-computer interfaces," in Proc. ISCA ITRW Speech Emotion, 2000, pp. 201-206.
71. J. Ang, R. Dhillon, E. Shriberg, and A. Stolcke, "Prosody-based automatic detection of annoyance and frustration in human-computer dialog," in Proc. Interspeech, Denver, CO, 2002, pp. 2037-2040.
72. C. M. Lee, S. Narayanan, and R. Pieraccini, "Combining acoustic and language information for emotion recognition," in Proc. ICSLP, 2002, pp. 873-1376
73. L. Devillers, L. Lamel, and I. Vasilescu, "Emotion detection in taskoriented spoken dialogs," in Proc. ICME, Baltimore, MD, 2003.
74. B. Schuller, R. Müller, M. Lang, and G. Rigoll, "Speaker independent emotion recognition by early fusion of acoustic and linguistic features within ensemble," in Proc. Interspeech, Lisbon, Portugal, 2005, pp. 805-808.
75. A. Batliner, S. Steidl, B. Schuller, D. Seppi, K. Laskowski, T. Vogt, L. Devillers, L. Vidrascu, N. Amir, L. Kessous, and V. Aharonson, "Combining efforts for improving automatic classification of emotional user states," in Proc. IS-LTC, Ljubliana, Slovenia, 2006, pp. 240-245.
76. T. Joachims, "Text categorization with support vector machines: Learning with many relevant features," in Proc. ECML, Chemniz, Germany, 1998, pp. 137-142.
77. B. Schuller, A. Batliner, S. Steidl, and D. Seppi, "Emotion recognition from speech: Putting asr in the loop," in Proc. ICASSP, Taipei, Taiwan, 2009, pp. 4585-4588.
78. B. Schuller, J. Schenk, and G. Rigoll, ""The Godfather" vs. "chaos": Comparing linguistic analysis based on online knowledge sources and bags-of-n-grams for movie review valence estimation," in Proc. ICDAR, Barcelona, Spain, 2009, pp. 858-862.
79. Y. Wang and I. H. Witten, "Modeling for optimal probability prediction," in Proc. 19th Int. Conf. Mach. Learn., Sydney, Australia, 2002, pp. 650-657.
80. M. Wöllmer, F. Eyben, B. Schuller, and G. Rigoll, "Robust vocabulary independent keyword spotting with graphical models," in Proc. ASRU, Merano, Italy, 2009, pp. 349-353.
81. J. A. Bilmes, "Graphical models and automatic speech recognition," in Mathematical Foundations of Speech and Language Processing, R. Rosenfeld, M. Ostendorf, S. Khudanpur, and M. Johnson, Eds. New York: Springer Verlag, 2003, pp. 191-246.
82. F. V. Jensen, An Introduction to Bayesian Networks. New York: Springer, 1996.
83. J. Bilmes and G. Zweig, "The graphical models toolkit: An open source software system for speech and time-series processing," in Proc. ICASSP, 2002, pp. 3916-3919.
84. J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren, "DARPA TIMIT Acoustic Phonetic Continuous Speech Corpus CDROM," 1993.
85. M. Wöllmer, F. Eyben, B. Schuller, E. Douglas-Cowie, and R. Cowie, "Data-driven clustering in emotional space for affect recognition using discriminatively trained LSTM networks," in Proc. Interspeech, Brighton, U.K., 2009, pp. 1595-1598.
86. M. Riedmiller and H. Braun, "A direct adaptive method for faster backpropagation learning: The RPROP algorithm," in Proc. IEEE Int. Conf. Neural Netw., 1993, pp. 586-591.
87. M. Grimm, K. Kroschel, and S. Narayanan, "Support vector regression for automatic recognition of spontaneous emotions in speech," in Proc. ICASSP, Honolulu, HI, 2007, pp. 1085-1088.
88. G. Castellano, L. Kessous, and G. Caridakis, "Emotion recognition through multiple modalities: Face, body gesture, speech," in Affect and Emotion in Human-Computer Interaction, C. Peter and R. Beale, Eds. New York: Springer, 2008, vol.4868, lNCS.
89. S. Bengio, "An asynchronous hidden Markov model for audio-visual speech recognition," Advances in NIPS 15 2003.
90. M. Wöllmer, M. Al-Hames, F. Eyben, B. Schuller, and G. Rigoll, "A multidimensional dynamic time warping algorithm for efficient multimodal fusion of asynchronous data streams," Neurocomputing, vol.73, pp. 366-380, 2009.