Analysis of emotionally salient aspects of fundamental frequency for emotion detection

IEEE Transactions on Audio, Speech and Language Processing

Volumn 17, Issue 4, 2009, Pages 582-596

  Busso, Carlos ^a   Lee, Sungbok ^a   Narayanan, Shrikanth ^a  

^a UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

Author keywords

Emotional speech analysis; Emotional speech recognition; Expressive speech; Intonation; Pitch contour analysis

Indexed keywords

EMOTIONAL SPEECH ANALYSIS; EMOTIONAL SPEECH RECOGNITION; EXPRESSIVE SPEECH; INTONATION; PITCH CONTOUR ANALYSIS;

INFORMATION THEORY; REGRESSION ANALYSIS; SPEECH ANALYSIS;

CONTINUOUS SPEECH RECOGNITION;

EID: 65249116503     PISSN: 15587916     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASL.2008.2009578     Document Type: Article

References (42)

1. R. W. Picard, "Affective Computing," MIT Media Laboratory Perceptual Computing Section, Cambridge, MA, USA, Tech. Rep. 321, Nov. 1995.
2. R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. Taylor, "Emotion recognition in human-computer interaction," IEEE Signal Process. Mag., vol. 18, no. 1, pp. 32-80, Jan. 2001.
3. A. Álvarez, I. Cearreta, J. López, A. Arruti, E. Lazkano, B. Sierra, and N. Garay, "Feature subset selection based on evolutionary algorithms for automatic emotion recognition in spoken spanish and standard basque language," in Proc. 9th Int. Conf. Text, Speech and Dialogue (TSD 2006), Brno, Czech Republic, Sep. 2006, pp. 565-572.
4. D. Ververidis and C. Kotropoulos, "Fast sequential floating forward selection applied to emotional speech features estimated on DES and SUSAS data collections," in Proc. XIV Eur. Signal Process. Conf. (EUSIPCO'06), Florence, Italy, Sep. 2006, pp. 929-932.
5. M. Sedaaghi, C. Kotropoulos, and D. Ververidis, "Using adaptive genetic algorithms to improve speech emotion recognition," in Proc. Int. Workshop Multimedia Signal Process. (MMSP'07), Chania, Crete, Greece, Oct. 2007, pp. 461-464.
6. C. Busso, S. Lee, and S. Narayanan, "Using neutral speech models for emotional speech analysis," in Proc. Interspeech'07 Eurospeech, Antwerp, Belgium, Aug. 2007, pp. 2225-2228.
7. E. Douglas-Cowie, L. Devillers, J. Martin, R. Cowie, S. Savvidou, S. Abrilian, and C. Cox, "Multimodal databases of everyday emotion: Facing up to complexity," in Proc. 9th Eur. Conf. Speech Commun. Technol. (Interspeech'05), Lisbon, Portugal, Sep. 2005, pp. 813-816.
8. P. Juslin and P. Laukka, "Communication of emotions in vocal expression and music performance: Different channels, same code?," Psychol. Bull., vol. 129, no. 5, pp. 770-814, Sep. 2003.
9. P. Lieberman and S. Michaels, "Some aspects of fundamental frequency and envelope amplitude as related to the emotional content of speech," J. Acoust. Soc. Amer., vol. 34, no. 7, pp. 922-927, Jul. 1962.
10. H. Wang, A. Li, and Q. Fang, "F0 contour of prosodic word in happy speech of Mandarin," in Affective Computing and Intelligent Interaction (ACII 2005), Lecture Notes in Artificial Intelligence 3784, J. Tao, T. Tan, and R. Picard, Eds. Berlin, Germany: Springer-Verlag, Nov. 2005, pp. 433-440.
11. A. Paeschke, M. Kienast, and W. Sendlmeier, "F0-contours in emotional speech," in Proc. 14th Int. Conf. Phonetic Sci. (ICPh'99), San Francisco, CA, Aug. 1999, pp. 929-932.
12. A. Paeschke, "Global trend of fundamental frequency in emotional speech," in Proc. Speech Prosody (SP'04), Nara, Japan, Mar. 2004, pp. 671-674.
13. T. Banziger and K. Scherer, "The role of intonation in emotional expressions," Speech Commun., vol. 46, no. 3-4, pp. 252-267, Jul. 2005.
14. M. Bulut and S. Narayanan, "On the robustness of overall F0-only modification effects to the perception of emotions in speech," J. Acoust. Soc. Amer., vol. 123, no. 6, pp. 4547-4558, Jun. 2008.
15. K. Scherer, D. Ladd, and K. Silverman, "Vocal cues to speaker affect: Testing two models," J. Acoust. Soc. Amer., vol. 76, no. 5, pp. 1346-1356, Nov. 1984.
16. C. Busso and S. Narayanan, "Interplay between linguistic and affective goals in facial expression during emotional utterances," in Proc. 7th Int. Seminar Speech Production (ISSP'06), Ubatuba-SP, Brazil, Dec. 2006, pp. 549-556.
17. G. Kochanski, "Prosody beyond fundamental frequency," in Methods in Empirical Prosody Research, ser. Language, Context and Cognition Series, D. Lenertova, R. Meyer, S. Pappert, P. Augurzky, I. Mleinek, N. Richter, and J. Schliesser, Eds. Berlin, Germany: Walter de Gruyter & Co., Apr. 2006, pp. 89-122.
18. E. Klabbers and J. V. Santen, "Clustering of foot-based pitch contours in expressive speech," in 5th ISCA Speech Synthesis Workshop, Pittsburgh, PA, Jun 2004, pp. 73-78.
19. D. Paul and J. Baker, "The design for the Wall Street Journal-based CSR corpus," in Proc. 2nd Int. Conf. Spoken Lang. Process. (ICSLP'92), Banff, AB, Canada, Oct. 1992, pp. 899-902.
20. J. Garofolo, L. Lamel, W. Fisher, J. Fiscus, D. Pallett, and N. Dahlgren, "TIMIT Acoustic-Phonetic Continuous Speech Corpus,' 1993.
21. S. Lee, S. Yildirim, A. Kazemzadeh, and S. Narayanan, "An articulatory study of emotional speech production," in Proc. 9th Eur. Conf. Speech Commun. Technol. (Interspeech'05 Eurospeech), Lisbon, Portugal, Sep. 2005, pp. 497-500.
22. M. Grimm, K. Kroschel, E. Mower, and S. Narayanan, "Primitives- based evaluation and estimation of emotions in speech," Speech Commun., vol. 49, no. 10-11, pp. 787-800, Oct.-Nov. 2007.
23. M. Liberman, K. Davis, M. Grossman, N. Martey, and J. Bell, "Emotional prosody speech and transcripts," in Proc. Linguist. Data Consortium, Philadelphia, PA, 2002, CD-ROM.
24. F. Burkhardt, A. Paeschke, M. Rolfes, W. Sendlmeier, and B. Weiss, "A database of German emotional speech," in 9th European Conf. Speech Communication and Technology (Interspeech'2005 Eurospeech), Lisbon, Portugal, Sep. 2005, pp. 1517-1520.
25. J. Montero, J. Gutiérrez-Arriola, S. Palazuelos, E. Enríquez, S. Aguilera, and J. Pardo, "Emotional speech synthesis: From speech database to TTS," in 5th Int. Conf. Spoken Lang. Process. (ICSLP'98), Sydney, Australia, Nov.-Dec. 1998, pp. 923-925.
26. E. Douglas-Cowie, N. Campbell, R. Cowie, and P. Roach, "Emotional speech: Towards a newgeneration of databases," Speech Commun., vol. 40, no. 1-2, pp. 33-60, Apr. 2003.
27. C. Busso and S. Narayanan, "Recording audio-visual emotional databases from actors: A closer look," in Proc. 2nd Int. Workshop Emotion: Corpora for Research on Emotion and Affect, Int. Conf. Language Resources and Evaluation (LREC 2008), Marrakech, Morocco, May 2008, pp. 17-22.
28. P. Boersma and D. Weeninck, "PRAAT, a system for doing phonetics by computer," Inst. Phon. Sci. Univ. of Amsterdam, Amsterdam, Netherlands, Tech. Rep. 132, 1996 [Online]. Available: http://www.praat.org.
29. K. Silverman, M. Beckman, J. Pitrelli, M. Ostendorf, C. Wightman, P. Price, J. Pierrehumbert, and J. Hirschberg, "ToBI: A standard for labelling english prosody," in 2th Int. Conf. Spoken Lang. Process. (ICSLP'92), Banff, AB, Canada, Oct. 1992, pp. 867-870.
30. S. Ananthakrishnan and S. Narayanan, "Automatic prosody labeling using acoustic, lexical, and syntactic evidence," IEEE Trans. Speech, Audio, Lang. Process., vol. 16, no. 1, pp. 216-228, Jan. 2008.
31. P. Taylor, "Analysis and synthesis of intonation using the tilt model," J. Acoust. Soc. Amer., vol. 107, no. 3, pp. 1697-1714, Mar. 2000.
32. E. Grabe, G.Kochanski, and J. Coleman, "Connecting intonation labels to mathematical descriptions of fundamental frequency," Lang. Speech, vol. 50, no. 3, pp. 281-310, Oct. 2007.
33. T. Cover and J. Thomas, Elements of Information Theory. New York: Wiley-Interscience, 2006.
34. R. Duda, P. Hart, and D. Stork, Pattern Classification. New York: Wiley-Interscience, 2000.
35. D. Hosmer and S. Lemeshow, Applied Logistic Regression, ser. Probability and Statistics. New York: Wiley, 2000.
36. N. Nagelkerke, "A note on a general definition of the coefficient of determination," Biometrika, vol. 78, no. 3, pp. 691-692, Sep. 1991.
37. C. Busso and S. Narayanan, "Joint analysis of the emotional fingerprint in the face and speech: A single subject study," in Proc. Int. Workshop Multimedia Signal Process. (MMSP 2007), Chania, Crete, Greece, Oct. 2007, pp. 43-47.
38. D. Ladd, K. Silverman, F. Tolkmitt, G. Bergmann, and K. Scherer, "Evidence for the independent function of intonation contour type, voice quality, and F0 range in signaling speaker affect," J. Acoust. Soc. Amer., vol. 78, no. 2, pp. 435-444, Aug. 1985.
39. S. Yildirim, M. Bulut, C. Lee, A. Kazemzadeh, C. Busso, Z. Deng, S. Lee, and S. Narayanan, "An acoustic study of emotions expressed in speech," in 8th Int. Conf. Spoken Lang. Process. (ICSLP'04), Jeju Island, Korea, Oct. 2004, pp. 2193-2196.
40. R. Cowie and R. Cornelius, "Describing the emotional states that are expressed in speech," Speech Commun., vol. 40, no. 1-2, pp. 5-32, Apr. 2003.
41. M. Grimm, K. Kroschel, and S. Narayanan, "The Vera AM Mittag German audio-visual emotional speech database," in IEEE Int. Conf. Multimedia Expo (ICME'08), Hannover, Germany, Jun. 2008, pp. 865-868.
42. T. Polzin, "Verbal and non-verbal cues in the communication of emotions," in Int. Conf. Acoust., Speech, Signal Process. (ICASSP'00), Istanbul, Turkey, Jun. 2000, pp. 53-59.