Classifier-based learning of nonlinear feature manifold for visualization of emotional speech prosody

IEEE Transactions on Affective Computing

Volumn 4, Issue 1, 2013, Pages 47-56

  Väyrynen, Eero ^a   Kortelainen, Jukka ^a   Seppänen, Tapio ^a  

^a UNIVERSITY OF OULU   (Finland)

Author keywords

Affect sensing and analysis; feature evaluation and selection; information visualization; machine learning; modeling human emotion; nonlinear manifold learning; speech analysis

Indexed keywords

AFFECT SENSING AND ANALYSIS; FEATURE EVALUATION AND SELECTION; HUMAN EMOTION; INFORMATION VISUALIZATION; NON-LINEAR MANIFOLD LEARNING;

INFORMATION SYSTEMS; LEARNING SYSTEMS; NEURAL NETWORKS; SPEECH ANALYSIS; VISUALIZATION;

DATA VISUALIZATION;

EID: 84876239811     PISSN: 19493045     EISSN: None     Source Type: Journal    
DOI: 10.1109/T-AFFC.2012.35     Document Type: Article

References (59)

1. C. Darwin, The Expression of the Emotions inMan and Animals. Univ. of Chicago Press, 1872/1965.
2. P. Ekman, "An Argument for Basic Emotions, " Cognition and Emotion, vol. 6, pp. 169-200, 1992.
3. C. E. Izard, "Basic Emotions, Relations among Emotions, and Emotion-Cognition Relations, " Psychological Rev. , vol. 99, pp. 561-565, 1992.
4. P. Ekman, "Basic Emotions, " Handbook of Cognition and Emotion, T. Dalgleish and M. Power, eds. pp. 45-60, John Wiley & Sons Inc, 1999.
5. J. A. Russell, "A Circumplex Model of Affect, " J. Personality and Social Psychology, vol. 39, pp. 1161-1178, 1980.
6. J. A. Russell and A. Mehrabian, "Evidence for a Three-Factor Theory of Emotions, " J. Research in Personality, vol. 11, no. 3, pp. 273-294, 1977.
7. C. A. Smith and P. C. Ellsworth, "Patterns of Cognitive Appraisal in Emotion, " J. Personality and Social Psychology, vol. 48, no. 4, pp. 813-838, 1985.
8. J. A. Russell and L. Feldman Barrett, "Core Affect, Prototypical Emotional Episodes, and Other Things Called Emotion: Dissecting the Elephant, " J. Personality and Social Psychology, vol. 76, pp. 805-819, 1999.
9. R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, and M. Schröder, "'Feeltrace': An Instrument for Recording Perceived Emotion in Real Time, " Proc. ISCA Tutorial and Research Workshop Speech and Emotion, pp. 19-24, Sept. 2000.
10. E. M. Daly, W. J. Lancee, and J. Polivy, "A Conical Model for the Taxonomy of Emotional Experience, " J. Personality and Social Psychology, vol. 45, pp. 443-457, 1983.
11. P. Laukka, P. Juslin, and R. Bresin, "A Dimensional Approach to Vocal Expression of Emotion, " Cognition and Emotion, vol. 19, no. 5, pp. 633-653, 2005.
12. R. Banse and K. R. Scherer, "Acoustic Profiles in Vocal Emotion Expression, " J. Personality and Social Psychology, vol. 70, no. 3, pp. 614-636, Mar. 1996.
13. K. R. Scherer, "Vocal Communication of Emotion: A Review of Research Paradigms, " Speech Comm. , vol. 40, nos. 1/2, pp. 227-256, 2003.
14. M. Schröder, R. Cowie, E. Douglas-Cowie, M. Westerdijk, and S. Gielen, "Acoustic Correlates of Emotion Dimensions in View of Speech Synthesis, " Proc. Seventh European Conf. Speech Comm. and Technology, vol. 1, pp. 87-90, 2001.
15. M. Schröder, "Emotional Speech Synthesis: A Review, " Proc. Seventh European Conf. Speech Comm. and Technology, vol. 1, pp. 561-564, 2001.
16. I. R. Murray and J. L. Arnott, "Implementation and Testing of a System for Producing Emotion-by-Rule in Synthetic Speech, " Speech Comm. , vol. 16, pp. 369-390, 1995.
17. D. Ververidis and C. Kotropoulos, "Emotional Speech Recognition: Resources, Features, and Methods, " Speech Comm. , vol. 48, no. 9, pp. 1162-1181, 2006.
18. K. R. Scherer, R. Banse, and H. Wallbott, "Emotion Inferences from Vocal Expression Correlate across Languages and Cultures, " J. Cross-Cultural Psychology, vol. 32, no. 1, pp. 76-92, 2001.
19. R. Cowie and R. R. Cornelius, "Describing the Emotional States That Are Expressed in Speech, " Speech Comm. , vol. 40, nos. 1/2, pp. 5-32, 2003.
20. E. Douglas-Cowie, N. Campbell, R. Cowie, and P. Roach, "Emotional Speech: Towards a New Generation of Databases, " Speech Comm. , vol. 40, no. 1, pp. 33-60, 2003.
21. F. Dellaert, T. S. Polzin, and A. Waibel, "Recognizing Emotion in Speech, " Proc. Int'l Congress Spoken Language Processing, vol. 3, pp. 1970-1973, Oct. 1996.
22. S. McGilloway, R. Cowie, E. Douglas-Cowie, S. Gielen, M. Westerdijk, and S. Stroeve, "Approaching Automatic Recognition of Emotion from Voice: A Rough Benchmark, " Proc. ISCA Tutorial and Research Workshop Speech and Emotion, pp. 207-212, Sept. 2000.
23. P. Oudeyer, "Novel Useful Features and Algorithms for the Recognition of Emotions in Human Speech, " Proc. Speech Prosody an Int'l Conf. , Apr. 2002.
24. L. tenBosch, "Emotions, Speech and the ASR Framework, " Speech Comm. , vol. 40, nos. 1/2, pp. 213-225, 2003.
25. J. Toivanen, E. Väyrynen, and T. Seppänen, "Automatic Discrimination of Emotion from Spoken Finnish, " Language and Speech, vol. 47, no. 4, pp. 383-412, 2004.
26. E. Väyrynen, J. Toivanen, and T. Seppänen, "Classification of Emotion in Spoken Finnish Using Vowel-Length Segments: Increasing Reliability with a Fusion Technique, " Speech Comm. , vol. 53, pp. 269-282, 2011.
27. R. Togneri, M. Alder, and Y. Attikiouzel, "Dimension and Structure of the Speech Space, " Proc. IEE I: Comm. , Speech and Vision, vol. 139, no. 2, pp. 123-127, Apr. 1992.
28. Z. Zeng, Z. Zhang, B. Pianfetti, J. Tu, and T. Huang, "Audio-Visual Affect Recognition in Activation-Evaluation Space, " Proc. IEEE Int'l Conf. Multimedia and Expo (ICME '05), p. 4, July 2005.
29. M. A. Nicolaou, H. Gunes, and M. Pantic, "Continuous Prediction of Spontaneous Affect from Multiple Cues and Modalities in Valence-Arousal Space, " IEEE Trans. Affective Computing, vol. 2, no. 2, pp. 92-105, Apr. -June 2011.
30. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. Taylor, "Emotion Recognition in Human-Computer Interaction, " IEEE Signal Processing Magazine, vol. 18, no. 1, pp. 32-80, Jan. 2001.
31. Z. Zeng, M. Pantic, G. Roisman, and T. Huang, "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions, " IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009.
32. R. A. Calvo and S. D'Mello, "Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications, " IEEE Trans. Affective Computing, vol. 1, no. 1, pp. 18-37, Jan. -June 2010.
33. S. T. Roweis and L. K. Saul, "Nonlinear Dimensionality Reduction by Locally Linear Embedding, " Science, vol. 290, no. 5500, pp. 2323-2326, 2000.
34. J. B. Tenenbaum, V. d. Silva, and J. C. Langford, "A Global Geometric Framework for Nonlinear Dimensionality Reduction, " Science, vol. 290, no. 5500, pp. 2319-2323, 2000.
35. L. J. J. van der Maaten, E. O. Postma, and H. J. van den Herik, "Dimensionality Reduction: A Comparative Review, " Tilburg University Technical Report, TiCC-TR 2009-005, http://homepage. tudelft. nl/19j49/Matlab-Toolbox-for-Dimensionality- Reduction-files/TR-Dimensiereductie. pdf, 2009.
36. B. Raytchev, I. Yoda, and K. Sakaue, "Head Pose Estimation by Nonlinear Manifold Learning, " Proc. 17th Int'l Conf. Pattern Recognition (ICPR '04), vol. 4, pp. 462-466, Aug. 2004.
37. M. Niskanen and O. Silvén, "Comparison of Dimensionality Reduction Methods for Wood Surface Inspection, " Proc. Sixth Int'l Conf. Quality Control by Artificial Vision, pp. 178-188, 2003.
38. M. Turtinen, M. Pietikäinen, and O. Silvén, "Visual Characterization of Paper Using Isomap and Local Binary Patterns, " Proc. IAPR Conf. Machine Vision Applications (IAPR MVA '05), pp. 210-213, May 2005.
39. X. Geng, D. -C. Zhan, and Z. -H. Zhou, "Supervised Nonlinear Dimensionality Reduction for Visualization and Classification, " IEEE Trans. Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 35, no. 6, pp. 1098-1107, Dec. 2005.
40. J. Kortelainen, E. Väyrynen, and T. Seppänen, "Isomap Approach to EEG-Based Assessment of Neurophysiological Changes during Anesthesia, " IEEE Trans. Neural Systems and Rehabilitation Eng. , vol. 19, no. 2, pp. 113-120, Apr. 2011.
41. M. You, C. Chen, J. Bu, J. Liu, and J. Tao, "Emotional Speech Analysis on Nonlinear Manifold, " Proc. 18th Int'l Conf. Pattern Recognition, vol. 3, pp. 91-94, 2006.
42. S. Zhang, B. Lei, A. Chen, C. Chen, and Y. Chen, "Kisomap-Based Feature Extraction for Spoken Emotion Recognition, " Proc. IEEE 10th Int'l Conf. Signal Processing (ICSP), pp. 1374-1377, Oct. 2010.
43. S. Zhang, L. Li, and Z. Zhao, "Speech Emotion Recognition Based on Supervised Locally Linear Embedding, " Proc. Int'l Conf. Comm. Circuits and Systems (ICCCAS), pp. 401-404, July 2010.
44. V. Jain and L. Saul, "Exploratory Analysis and Visualization of Speech and Music by Locally Linear Embedding, " Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing (ICASSP '04), vol. 3, pp. iii-984-7, May 2004.
45. J. Kim, S. Lee, and S. Narayanan, "An Exploratory Study of Manifolds of Emotional Speech, " Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing (ICASSP), pp. 5142-5145, Mar. 2010.
46. T. Seppänen, J. Toivanen, and E. Väyrynen, "MediaTeam Speech Corpus: A First Large Finnish Emotional Speech Database, " Proc. 15th Int'l Congress of Phonetic Sciences, vol. 3, pp. 2469-2472, 2003.
47. E. Väyrynen, "Automatic Emotion Recognition from Speech, " Master's thesis, Univ. of Oulu, 2005.
48. M. Balasubramanian and E. L. Schwartz, "The Isomap Algorithm and Topological Stability, " Science, vol. 295, no. 5552, p. 7, 2002.
49. M. Vlachos, C. Domeniconi, D. Gunopulos, G. Kollios, and N. Koudas, "Non-Linear Dimensionality Reduction Techniques for Classification and Visualization, " Proc. Eighth ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, pp. 645-651, 2002.
50. H. Choi and S. Choi, "Kernel Isomap, " Electronics Letters, vol. 40, no. 25, pp. 1612-1613, Dec. 2004.
51. R. -J. Gu and W. -B. Xu, "Weighted Kernel Isomap for Data Visualization and Pattern Classification, " Computational Intelligence and Security, Y. Wang, Y. -M. Cheung, and H. Liu, eds. , pp. 1050-1057, Springer, 2007.
52. P. Pudil, J. Novovicová, and J. Kittler, "Floating Search Methods in Feature Selection, " Pattern Recognition Letters, vol. 15, pp. 1119-1125, 1994.
53. M. Hubert, P. J. Rousseeuw, and K. V. Branden, "ROBPCA: A New Approach to Robust Principal Component Analysis, " Technometrics, vol. 47, pp. 64-79, 2005.
54. K. R. Scherer, "Vocal Affect Expression: A Review and a Model for Future Research, " Psychological Bull. , vol. 99, no. 2, pp. 143-165, 1986.
55. J. Ang, R. Dhillon, A. Krupski, E. Shriberg, and A. Stolcke, "Prosody-Based Automatic Detection of Annoyance and Frustration in Human-Computer Dialog, " Proc. Seventh Int'l Confrence on Spoken Language Processing, pp. 2037-2040, Sept. 2002.
56. A. Batliner, K. Fischer, R. Huber, J. Spilker, and E. Nöth, "Desperately Seeking Emotions or: Actors, Wizards, and Human Beigns, " Proc. ISCA Tutorial and Research Workshop Speech and Emotion, pp. 195-200, Sept. 2000.
57. T. Johnstone and K. R. Scherer, "Vocal Communication of Emotion, " Handbook of Emotion, M. Lewis and J. Haviland, eds. , pp. 220-235, The Guilford Press, 2000.
58. A. M. Laukkanen, E. Vilkman, P. Alku, and H. Oksanen, "Physical Variations Related to Stress and Emotional State: A Preliminary Study, " J. Phonetics, vol. 24, pp. 313-335, 1996.
59. A. M. Laukkanen, E. Vilkman, P. Alku, and H. Oksanen, "On the Perception of Emotions in Speech: The Role of Voice Quality, " Locopedics, Phonatrics and Vocology, vol. 22, no. 4, pp. 157-168, 1997.