Subjective responses to synthesised speech with lexical emotional content: The effect of the naturalness of the synthetic voice

Behaviour and Information Technology

Volumn 32, Issue 2, 2013, Pages 117-131

  Ilves, Mirja ^a   Surakka, Veikko ^a  

^a TAMPERE UNIVERSITY OF TECHNOLOGY   (Finland)

Author keywords

emotion; naturalness; synthesised speech

Indexed keywords

AFFECTIVE WORDS; DIPHONES; EMOTION; EMOTIONAL COMMUNICATIONS; EMOTIONAL EXPERIENCES; FORMANT SYNTHESIS; NATURALNESS; POSITIVE FEELINGS; SUBJECTIVE RATING; SYNTHESIS TECHNIQUES; SYNTHESISED; SYNTHETIC VOICES; UNIT SELECTION; VOICE QUALITY;

HUMAN COMPUTER INTERACTION; SOCIAL SCIENCES;

SPEECH SYNTHESIS;

EID: 84874406163     PISSN: 0144929X     EISSN: 13623001     Source Type: Journal    
DOI: 10.1080/0144929X.2012.702285     Document Type: Review

References (76)

1. Alais, D., Morrone, C., and, Burr, D. 2006. Separate attentional resources for vision and audition. Proceedings of the Royal Society, 273 (1592): 1339-1345. (doi:10.1098/rspb.2005.3420) (Pubitemid 47308753)
2. Anttonen, J., and, Surakka, V. Emotions and heart rate while sitting on a chair. Proceedings of the CHI 2005 SIGCHI conference on human factors in computing systems. April 2-7. pp. 491-499. New York: ACM.
3. Aula, A., and, Surakka, V. Auditory emotional feedback facilitates human-computer interaction. Proceedings of the HCI 2002 international conference on human-computer interaction. September 2-6. pp. 337-349. London: Springer-Verlag.
4. Bartneck, C. 2001. How convincing is Mr Data's smile: affective expressions of machines. User Modeling and User-Adapted Interaction, 11 (4): 279-295. (doi:10.1023/A:1011811315582) (Pubitemid 32902624)
5. Beale, R., and, Creed, C. 2009. Affective interaction: how emotional agents affect users. International Journal of Human-Computer Studies, 67 (9): 755-776. (doi:10.1016/j.ijhcs.2009.05.001)
6. Bertels, J., Kolinsky, R., and, Morais, J. 2009. Norms of emotional valence, arousal, threat value and shock value for 80 spoken French words: comparison between neutral and emotional tones of voice. Psychologica Belgica, 49 (1): 19-40.
7. Bradley, M. M., and, Lang, P. J. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of Behavioral Therapy and Experimental Psychiatry, 25 (1): 49-59. (doi:10.1016/0005-7916(94) 90063-9) (Pubitemid 24214918)
8. Bradley, M. M., and, Lang, P. J. 1999. " Affective norms for English words (ANEW): stimuli, instruction manual and affective ratings ". In Technical report C-1, Gainesville: University of Florida.
9. Bradley, M. M., Lang, P. J., and, Cuthbert, B. N. 1993. Emotion, novelty, and the startle reflex: habituation in humans. Behavioral Neuroscience, 107 (6): 970-980. (doi:10.1037/0735-7044.107.6.970) (Pubitemid 24018207)
10. Brave, S., Nass, C., and, Hutchinson, K. 2005. Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent. International Journal of Human-Computer Studies, 62 (2): 161-178. (doi:10.1016/j.ijhcs.2004.11.002) (Pubitemid 40206241)
11. Breiter, H. C. 1996. Response and habituation of the human amygdala during visual processing of facial expression. Neuron, 17 (5): 875-887. (doi:10.1016/S0896-6273(00)80219-6) (Pubitemid 26424015)
12. Buchanan, T. W. 2006. The influence of autonomic arousal and semantic relatedness on memory for emotional words. International Journal of Psychophysiology, 61 (1): 26-33. (doi:10.1016/j.ijpsycho.2005.10.022) (Pubitemid 43947277)
13. Cahn, J. 1990. The generation of affect in synthesized speech. Journal of the American Voice I/O Society, 8 (1): 1-19.
14. Calvo, R. A., and, D'Mello, S. 2010. Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1 (1): 18-37. (doi:10.1109/T-AFFC.2010.1)
15. Canli, T. 1998. Hemispheric asymmetry for emotional stimuli detected with fMRI. Neuroreport, 9 (14): 3233-3239. (doi:10.1097/00001756-199810050-00019) (Pubitemid 28522342)
16. Christie, I. C., and, Friedman, B. H. 2004. Autonomic specificity of discrete emotion and dimensions of affective space: a multivariate approach. International Journal of Psychophysiology, 51 (2): 143-153. (doi:10.1016/j. ijpsycho.2003.08.002) (Pubitemid 38018963)
17. Conati, C., and, Maclaren, H. 2009. Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction, 19 (3): 267-303. (doi:10.1007/s11257-009-9062-8)
18. Cowie, R., and, Cornelius, R. R. 2003. Describing the emotional states that are expressed in speech. Speech Communication, 40 (1-2): 5-32. (doi:10.1016/S0167-6393(02)00071-7)
19. Dai, L. 2005. Speech-based cursor control using grids: modeling performance and comparisons with other solutions. Behaviour & Information Technology, 24 (3): 219-230. (doi:10.1080/01449290412331328563) (Pubitemid 41673235)
20. Dehn, D. M., and, van Mulken, S. 2000. The impact of animated interface agents: a review of empirical research. International Journal of Human-Computer Studies, 52 (1): 1-22. (doi:10.1006/ijhc.1999.0325) (Pubitemid 30558463)
21. Dutoit, T. 1997. An introduction to text-to-speech synthesis, Dordrecht: Kluwer Academic Publisher.
22. Ekman, P. 1992. An argument for basic emotions. Cognition and Emotion, 6 (3/4): 169-200.
23. Elliot, A. 1999. Approach and avoidance motivation and achievement goals. Educational Psychologist, 34 (3): 169-189. (doi:10.1207/s15326985ep3403-3)
24. Estrada, A. E., Isen, A. M., and, Young, M. J. 1994. Positive affect improves creative problem solving and influences reported source of practice satisfaction in physicians. Motivation and Emotion, 18 (4): 285-299. (doi:10.1007/BF02856470)
25. Fredrickson, B. L., and, Branigan, C. 2005. Positive emotions broaden the scope of attention and thought-action repertoires. Cognition and Emotion, 19 (3): 313-332. (doi:10.1080/02699930441000238) (Pubitemid 40542863)
26. Frijda, N. H. 1988. The laws of emotion. American Psychologist, 43 (5): 349-358. (doi:10.1037/0003-066X.43.5.349)
27. Grühn, D., and, Scheibe, S. 2008. Age-related differences in valence and arousal ratings of pictures from the International Affective Picture System (IAPS): do ratings become more extreme with age?. Behavior Research Methods, 40 (2): 512-521. (doi:10.3758/BRM.40.2.512)
28. Guthrie, S. E. 1993. Faces in the clouds: a new theory of religion, New York: Oxford University Press.
29. Hassenzahl, M., and, Tractinsky, N. 2006. User experience-a research agenda. Behavior & Information Technology, 25 (2): 91-97. (doi:10.1080/01449290500330331) (Pubitemid 43065340)
30. Harmon-Jones, E., and, Siegelman, J. 2001. State anger and prefrontal brain activity: evidence that insult-related relative left-prefrontal activation is associated with experienced anger and aggression. Journal of Personality and Social Psychology, 80 (5): 797-803. (doi:10.1037/0022-3514.80.5.797) (Pubitemid 33594893)
31. Iida, A. 2003. A corpus-based speech synthesis system with emotion. Speech Communication, 40 (1-2): 161-187. (doi:10.1016/S0167-6393(02)00081-X)
32. Ilves, M., and, Surakka, V. Subjective and physiological responses to emotional content of synthesized speech. Proceedings of the CASA 2004 international conference on computer animation and social agents. July 7-9. Edited by: Magnenat-Thalmann, N., Joslin, C. and Kim, H. pp. 19-26. Geneva: Computer Graphics Society.
33. Ilves, M., and, Surakka, V. 2009. " Emotions, anthropomorphism of speech synthesis, and psychophysiology ". In Emotions in the human voice. Volume III: culture and perception, Edited by: Izdebski, K. 137-152. San Diego: Plural Publishing Inc.
34. Ilves, M., Surakka, V., and, Vanhala, T. 2011. " The effects of emotionally worded synthesized speech on the ratings of emotions and voice quality ". In ACII 2011, Part I, lecture notes in computer science, 6974, Edited by: D'Mello, S. 588-598. Berlin Heidelberg: Springer-Verlag.
35. Isen, A. M., Daubman, K. A., and, Nowicki, G. P. 1987. Positive affect facilitates creative problem solving. Journal of Personality and Social Psychology, 52 (6): 1122-1131. (doi:10.1037/0022-3514.52.6.1122)
36. Kappas, A., Hess, U., and, Scherer, K. R. 1991. " Voice and emotion ". In Fundamentals of nonverbal behavior. Studies in emotion & social interaction, Edited by: Feldman, R. S. and Rimé, B. 201-238. Cambridge: Cambridge University Press.
37. Klein, J., Moon, Y., and, Picard, R. W. 2002. This computer responds to user frustration: theory, design, and results. Interacting with Computers, 14 (2): 119-140. (doi:10.1016/S0953-5438(01)00053-4) (Pubitemid 34199155)
38. Knoll, M. A., Uther, M., and, Costall, A. 2011. Using the Internet for speech research: an evaluative study examining affect in speech. Behaviour and Information Technology,. [online]. Available from: http://www.tandfonline.com/ doi/abs/10.1080/0144929X.2011.577192 [Accessed 29 August 2011]
39. Ku, J. 2005. Experimental results of affective valence and arousal to avatar's facial expressions. Cyberpsychology & Behavior, 8 (5): 493-503. (doi:10.1089/cpb.2005.8.493) (Pubitemid 41540933)
40. Laird, J. D. 1982. Remembering what you feel: effects of emotion on memory. Journal of Personality and Social Psychology, 42 (4): 646-657. (doi:10.1037/0022-3514.42.4.646)
41. Lang, P. J., Bradley, M. M., and, Cuthbert, B. N. 1992. A motivational analysis of emotion: reflex-cortex connections. Psychological Science, 3 (1): 44-49. (doi:10.1111/j.1467-9280.1992.tb00255.x)
42. Lang, P. J. 1993. Looking at pictures: affective, facial, visceral, and behavioral reactions. Psychophysiology, 30 (3): 261-273. (doi:10.1111/j.1469- 8986.1993.tb03352.x) (Pubitemid 23108336)
43. Larsen, J. T., Norris, C. J., and, Cacioppo, J. T. 2003. Effects of positive and negative affect on electromyographic activity over zygomaticus major and corrugator supercilii. Psychophysiology, 40 (5): 776-785. (doi:10.1111/1469-8986.00078) (Pubitemid 37128041)
44. Law, E. L.-C., and, van Schaik, P. 2010. Modeling user experience-An agenda for research and practice. Interacting with Computers, 22 (5): 313-322. (doi:10.1016/j.intcom.2010.04.006)
45. Mauss, I. B., and, Robinson, M. D. 2009. Measures of emotion: a review. Cognition and Emotion, 23 (2): 209-237. (doi:10.1080/02699930802204677)
46. Miller, J. D. 1974. Effects of noise on people. Journal of the Acoustical Society of America, 56 (3): 729-764. (doi:10.1121/1.1903322)
47. Minnema, M. T., and, Knowlton, B. J. 2008. Directed forgetting of emotional words. Emotion, 8 (5): 643-652. (doi:10.1037/a0013441)
48. Mori, M. 1970. Bukimi no tani (The uncanny valley). Energy, 7 (4): 33-35. (MacDorman, K.F. and Minato, T., Translated)
49. Murray, I. R., and, Arnott, J. L. 1993. Toward the simulation of emotion in synthetic speech: a review of the literature on human vocal emotion. Journal of Acoustical Society of America, 93 (2): 1097-1108. (doi:10.1121/1.405558) (Pubitemid 23059837)
50. Murray, I. R., and, Arnott, J. L. 1995. Implementation and testing of a system for producing emotion-by-rule in synthetic speech. Speech Communication, 16 (4): 369-390. (doi:10.1016/0167-6393(95)00005-9)
51. Nass, C., and, Brave, S. 2005. Wired for speech: how voice activates and advances the human-computer relationship, Cambridge: The MIT Press.
52. Nass, C., and, Lee, K. M. 2001. Does computer-synthesized speech manifest personality? Experimental tests of recognition, similarity-attraction, and consistency-attraction. Journal of Experimental Psychology: Applied, 7 (3): 171-181. (doi:10.1037/1076-898X.7.3.171) (Pubitemid 33595084)
53. Nass, C., Steuer, J., and, Tauber, E. R. Computers are social actors. Proceedings of the CHI 1994 SIGCHI conference on human factors in computing systems. April 24-28. pp. 72-78. New York: ACM.
54. Nowak, K. L. 2004. The influence of anthropomorphism and agency on social judgment in virtual environments. Journal of Computer-Mediated Communication [online], 9. Available from: http://www3.interscience.wiley.com/cgi-bin/ fulltext/120837918/HTMLSTART [Accessed 3 June 2011]
55. Nowak, K. L., and, Rauh, C. 2005. The influence of the avatar on online perceptions of anthropomorphism, androgyny, credibility, homophily, and attraction. Journal of Computer-Mediated Communication, 11 (1): 153-178. (doi:10.1111/j.1083-6101.2006.tb00308.x)
56. Nowak, K. L., and, Rauh, C. 2008. Choose your "buddy icon" carefully: the influence of avatar androgyny, anthropomorphism and credibility in online interactions. Computers in Human Behavior, 24 (4): 1473-1493. (doi:10.1016/j.chb.2007.05.005) (Pubitemid 351551476)
57. Partala, T., and, Surakka, V. 2004. The effects of affective interventions in human-computer interaction. Interacting with Computers, 16 (2): 295-309. (doi:10.1016/j.intcom.2003.12.001)
58. Picard, R. W. 1995. Affective computing, Massachusetts: The MIT Press.
59. Prendinger, H., Mori, J., and, Ishizuka, M. 2005. Using human physiology to evaluate subtle expressivity of a virtual quizmaster in a mathematical game. International Journal of Human-Computer Studies, 62 (2): 231-245. (doi:10.1016/j.ijhcs.2004.11.009) (Pubitemid 40206245)
60. Reeves, B., and, Nass, C. 1996. The media equation: how people treat computers, television, and new media like real people and places, New York: Cambridge University Press.
61. Saarni, T. 2010. Segmental durations of speech, Thesis (PhD). University of Turku. Available from: https://www.doria.fi/handle/10024/52552 [Accessed 11 January 2012]
62. Scharf, B. 1998. " Auditory attention: the psychoacoustical approach ". In Attention, Edited by: Pashler, H. 75-117. Hove: Psychology Press.
63. Schneider, W., Eschman, A., and, Zuccolotto, A. 2002. E-prime user's guide, Pittsburgh: Psychology software Tools Inc.
64. Schröder, M. 2009. " Approaches to emotional expressivity in synthetic speech ". In Emotions in the human voice. Volume III: culture and perception, Edited by: Izdebski, K. 307-321. San Diego: Plural Publishing Inc.
65. Shi, Y., and, Zhou, L. 2009. Supporting dictation speech recognition error correction: the impact of external information. Behaviour and Information Technology,. [online]. Available from: http://www.tandfonline.com/doi/abs/10. 1080/01449290903353039 [Accessed 30 August 2011]
66. Shneiderman, B., and, Maes, P. 1997. Direct manipulation vs. interface agents. Interactions, 4 (6): 42-61. (doi:10.1145/267505.267514)
67. Sjölander, K., and, Beskow, J. Wavesurfer-an open source speech tool. Proceedings of the ICSLP 2000 international conference on spoken language processing. October 16-20. pp. 464-467. Beijing, China
68. Spering, M., Wagener, D., and, Funke, J. 2005. The role of emotions in complex problem-solving. Cognition and Emotion, 19 (8): 1252-1261. (doi:10.1080/02699930500304886)
69. Surakka, V., and, Hietanen, J. K. 1998. Facial and emotional reactions to Duchenne and non-Duchenne smiles. International Journal of Psychophysiology, 29 (1): 23-33. (doi:10.1016/S0167-8760(97)00088-3) (Pubitemid 28243134)
70. Viswanathan, M., and, Viswanathan, M. 2005. Measuring speech quality for text-to-speech systems: development and assessment of a modified mean opinion score (MOS) scale. Computer Speech and Language, 19 (1): 55-83. (doi:10.1016/j.csl.2003.12.001)
71. Waaramaa-Mäki-Kulmala, T. 2009. Emotions in voice; Acoustic and perceptual analysis of voice quality in the vocal expression of emotions, Thesis (PhD). University of Tampere. Available from: http://acta.uta.fi/pdf/978-951- 44-7667-9.pdf [Accessed 1 September 2011]
72. Wambacq, I. J., Shea-Miller, K. J., and, Abubakr, A. 2004. Non-voluntary and voluntary processing of emotional prosody: an event-related potentials study. Cognitive Neuroscience and Neuropsychology, 15 (3): 555-559. (Pubitemid 38328159)
73. Werner, N. S., Duschek, S., and, Schandry, R. 2009. Relationship between affective states and decision-making. International Journal of Psychophysiology, 74 (3): 259-265. (doi:10.1016/j.ijpsycho.2009.09.010)
74. Weyers, P. 2006. Electromyographic responses to static and dynamic avatar emotional facial expressions. Psychophysiology, 43 (5): 450-453. (doi:10.1111/j.1469-8986.2006.00451.x) (Pubitemid 44370047)
75. Wickens, C. D. 2002. Multiple resources and performance prediction. Theoretical Issues in Ergonomics Science, 3 (2): 159-177. (doi:10.1080/ 14639220210123806)
76. Zajonc, R. B. 1980. Feeling and thinking: preferences need no inferences. American Psychologist, 35 (2): 151-175. (doi:10.1037/0003-066X.35.2.151)