Real-time recognition of affective states from nonverbal features of speech and its application for public speaking skill analysis

IEEE Transactions on Affective Computing

Volumn 2, Issue 2, 2011, Pages 66-78

  Pfister, Tomas ^a   Robinson, Peter ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Affect analysis; emotion in human computer interaction.; public speaking; speech analysis; speech coaching

Indexed keywords

AFFECT ANALYSIS; AFFECTIVE STATE; CLASSIFICATION ACCURACY; CLASSIFICATION ALGORITHM; CROSS VALIDATION; EMOTION IN HUMAN-COMPUTER INTERACTION.; MIND READING; NON-VERBAL FEATURES; NOVEL APPLICATIONS; PUBLIC SPEAKING; REAL TIME RECOGNITION; REAL-TIME INFERENCE;

SPEECH ANALYSIS;

INFERENCE ENGINES;

EID: 84857825591     PISSN: 19493045     EISSN: None     Source Type: Journal    
DOI: 10.1109/T-AFFC.2011.8     Document Type: Article

References (47)

1. K.R. Scherer, "Vocal Affect Expression: A Review and a Model for Future Research," Psychological Bull., vol. 99, pp. 143-165, 1986.
2. R. Picard, "Affective Computing: From Laughter to IEEE," IEEE Trans. Affective Computing, vol. 1, no. 1, pp. 11-17, Jan.-June 2010.
3. R. Picard, Affective Computing. The MIT Press, 2000.
4. J.D. Haynes and G. Rees, "Decoding Mental States from Brain Activity in Humans," Nature Rev. Neuroscience, vol. 7, pp. 523-534, 2006. (Pubitemid 43989773)
5. T. Sobol Shikler, "Analysis of Affective Expression in Speech," PhD dissertation, Cambridge Univ., 2007.
6. T. Pfister and P. Robinson, "Speech Emotion Classification and Public Speaking Skill Assessment," Proc. Int'l Workshop Human Behavior Understanding in conjunction with the Int'l Conf. Pattern Recognition, pp. 151-162, 2010.
7. E. Uldall, "Attitudinal Meanings Conveyed by Intonation Contours," Language and Speech, vol. 3, no. 4, pp. 223-234, Oct./Dec. 1960.
8. G.L. Huttar, "Relations between Prosodic Variables and Emotions in Normal American English Utterances," J. Acoustical Soc. of Am., vol. 41, pp. 1581-1581, 1967.
9. F. Dellaert, T. Polzin, and A. Waibel, "Recognizing Emotion in Speech," Proc. Fourth Int'l Conf. Spoken Language Processing, vol. 3, pp. 1970-1973, 1996.
10. A. Batliner, K. Fisher, R. Huber, J. Spilker, and E. Noth, "Desperately Seeking Emotions or: Actors, Wizards, and Human Beings," Proc. Int'l Speech Comm. Assoc. Workshop Speech and Emotion, pp. 195-200, 2000.
11. K. Forbes-Riley and D. Litman, "Predicting Emotion from Spoken Dialogue from Multiple Knowledge Sources," Proc. Human Language Technology Conf. North Am. Chapter of the Assoc. for Computational Linguistics, 2004.
12. S. Steidl, M. Levit, A. Batliner, E. Noeth, and E. Niemann, "Of All Things the Measure Is Man-Automatic Classification of Emotions and Interlabeler Consistency," Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, 2005.
13. C. Lee, S. Narayanan, and R. Pieraccini, "Combining Acoustic and Language Information for Emotion Recognition," Proc. Int'l Conf. Spoken Language Processing, pp. 873-876, 2002.
14. R. Craggs and M. Wood, "A Two-Dimensional Annotation Scheme for Emotion in Dialogue," Proc. Am. Assoc. for Artificial Intelligence Spring Symp. Exploring Attitude and Affect in Text: Theories and Applications, 2004.
15. B. Schuller and G. Rigoll, "Recognising Interest in Conversational Speech-Comparing Bag of Frames and Supra-Segmental Features," Proc. 10th Ann. Conf. Int'l Speech Comm. Assoc., 2009.
16. B. Schuller, S. Steidl, and A. Batliner, "The Interspeech 2009 Emotion Challenge," Proc. 10th Ann. Conf. Int'l Speech Comm. Assoc. 2009.
17. C. Lee, E. Mower, C. Busso, S. Lee, and S. Narayanan, "Emotion Recognition Using a Hierarchical Binary Decision Tree Approach," Proc. 10th Ann. Conf. Int'l Speech Comm. Assoc., 2009.
18. P. Dumouchel, N. Dehak, Y. Attabi, R. Dehak, and N. Boufaden, "Cepstral and Long-Term Features for Emotion Recognition," Proc. 10th Ann. Conf. Int'l Speech Comm. Assoc., 2009.
19. E. Douglas-Cowie, N. Campbell, R. Cowie, and P. Roach, "Emotional Speech: Towards a New Generation of Databases," Speech Comm., vol. 40, nos. 1/2, pp. 33-60, 2003.
20. S. Abrilian, L. Devillers, S. Buisine, and J. Martin, "EmoTV1: Annotation of Real-Life Emotions for the Specification of Multimodal Affective Interfaces," Proc. 11th Int'l Conf. Human-Computer Interaction, 2005.
21. F. Burkhardt, A. Paeschke, M. Rolfes, W. Sendlmeier, and B. Weiss, "A Database of German Emotional Speech," Proc. Ninth European Conf. Speech Comm. and Technology, pp. 1517-1520, 2005. (Pubitemid 43908362)
22. O. Martin, I. Kotsia, B. Macq, and I. Pitas, "The eNTERFACE05 Audio-Visual Emotion Database," Proc. 22nd Int'l Conf. Data Eng. Workshops, 2006.
23. B. Schuller, R. Müller, F. Eyben, J. Gast, B. Hörnler, M. Wöllmer, G. Rigoll, A. Hörthker, and H. Konosu, "Being Bored? Recognising Natural Interest by Extensive Audiovisual Integration for Real-Life Application," Image and Vision Computing, vol. 27, no. 12, pp. 1760-1774, 2009.
24. S. Baron-Cohen, O. Golan, S. Wheelwright, and J.J. Hill, Mind Reading: The Interactive Guide to Emotions. Jessica Kingsley Publishers, 2004.
25. T. Sobol Shikler and P. Robinson, "Classification of Complex Information: Inference of Co-Occurring Affective States from Their Expressions in Speech," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 7, pp. 1284-1297, July 2010.
26. F. Eyben, M. Wöllmer, and B. Schuller, "openEAR-Introducing the Munich Open-Source Emotion and Affect Recognition Toolkit," Proc. Third Int'l Conf. Affective Computing and Intelligent Interaction, Sept. 2009.
27. B. Schuller, S. Reiter, R. Müller, M. Al-Hames, M. Lang, and G. Rigoll, "Speaker Independent Speech Emotion Recognition by Ensemble Classification," Proc. IEEE Int'l Conf. Multimedia and Expo, 2005.
28. S. Scherer, F. Schwenker, and G. Palm, "Classifier Fusion for Emotion Recognition from Speech," Advanced Intelligent Environments, pp. 95-117, Springer, 2009.
29. B. Schuller, R. Müller, M. Lang, and G. Rigoll, "Speaker Independent Emotion Recognition by Early Fusion of Acoustic and Linguistic Features within Ensembles," Proc. Ninth European Conf. Speech Comm. and Technology-Interspeech '05, 2005.
30. J. Russell, J. Bachorowski, and J. Fernandez-Dols, "Facial and Vocal Expressions of Emotion," Ann. Rev. of Psychology, vol. 54, pp. 329-350, 2003.
31. K. Scherer, "Vocal Communication of Emotion: A Review of Research Paradigms," Speech Comm., vol. 40, nos. 1/2, pp. 227-256, 2003.
32. K. Scherer, T. Johnstone, and G. Klasmeyer, "Vocal Expression of Emotion," Handbook of Affective Sciences, pp. 433-456, Oxford Univ. Press, 2003.
33. R. Barr, B. Hopkins, and J. Green, Crying as a Sign, a Symptom, and a Signal. Mac Keith Press, 2000.
34. R. Cowie and R. Cornelius, "Describing the Emotional States that are Expressed in Speech," Speech Comm., vol. 40, nos. 1/2, pp. 5-32, 2003.
35. R. Calvo and S. D'Mello, "Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications," IEEE Trans. Affective Computing, vol. 1, no. 1, pp. 18-37, Jan.-June 2010.
36. R. Reisenzein, "Broadening the Scope of Affect Detection Research," IEEE Trans. Affective Computing, vol. 1, no. 1, pp. 42-45, Jan.-June 2010.
37. O. Golan, S. Baron-Cohen, S. Wheelwright, and J.J. Hill, "Systemizing Empathy: Teaching Adults with Asperger Syndrome and High Functioning Autism to Recognize Complex Emotions Using Interactive Multimedia," Development and Psychopathology, vol. 18, pp. 589-615, 2006.
38. V.N. Vapnik, The Nature of Statistical Learning Theory. Springer, 1998.
39. M. Berouti, R. Schwartz, and J. Makhoul, "Enhancement of Speech Corrupted by Acoustic Noise," Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, pp. 208-211, 1979. (Pubitemid 9454996)
40. G. Yu, S. Mallat, and E. Bacry, "Audio Denoising by Time-Frequency Block Thresholding," IEEE Trans. Signal Processing, vol. 56, no. 5, pp. 1830-1839, May 2008.
41. H. Altun and G. Polat, "New Frameworks to Boost Feature Selection Algorithms in Emotion Detection for Improved Human-Computer Interaction," Proc. Second Int'l Symp. Advances in Brain, Vision, and Artificial Intelligence, pp. 533-541, 2007.
42. M.A. Hall and L.A. Smith, "Feature Selection for Machine Learning: Comparing a Correlation-Based Filter Approach to the Wrapper," Proc. Conf. Florida Artificial Intelligence Symp., 1999.
43. L. Qing-kun and Q. Pei-wen, "Model Selection for SVM Using Mutative Scale Chaos Optimization Algorithm," J. Shanghai Univ., vol. 10, pp. 531-534, 2006.
44. R. Fan, P. Chen, and C. Lin, "Working Set Selection Using Second Order Information for Training SVM," J. Machine Learning Research, vol. 6, pp. 1889-1918, 2005. (Pubitemid 41798130)
45. S. Pöyhönen, A. Arkkio, P. Jover, and H. Hyötyniemi, "Coupling Pairwise Support Vector Machines for Fault Classification," Control Eng. Practice, vol. 13, no. 6, pp. 759-769, 2005. (Pubitemid 40234110)
46. P. Somol, J. Novovicová , J. Grim, and P. Pudil, "Dynamic Oscillating Search Algorithm for Feature Selection," Proc. Int'l Conf. Pattern Recognition, pp. 1-4, 2009.
47. P. Somol and P. Pudil, "Oscillating Search Algorithms for Feature Selection," Proc. Int'l Conf. Pattern Recognition, pp. 406-409, 2000.