The MEI robot: Towards using motherese to develop multimodal emotional intelligence

IEEE Transactions on Autonomous Mental Development

Volumn 6, Issue 2, 2014, Pages 126-138

  Lim, Angelica ^a   Okuno, Hiroshi G ^a  

^a KYOTO UNIVERSITY   (Japan)

Author keywords

Cross modal recognition; emotion recognition; gait; gaussian mixture; gesture; motherese; SIRE; voice

Indexed keywords

INTELLIGENT ROBOTS; PATTERN RECOGNITION; ROBOTS; SPEECH;

CROSS-MODAL; EMOTION RECOGNITION; GAIT; GAUSSIAN MIXTURES; GESTURE; MOTHERESE; SIRE;

SPEECH RECOGNITION;

EID: 84903213220     PISSN: 19430604     EISSN: None     Source Type: Journal    
DOI: 10.1109/TAMD.2014.2317513     Document Type: Article

References (79)

1. K. Amaya and A. Bruderlin et al., "Emotion from motion," Graphics Interface, vol. 96, pp. 222-229, 1996.
2. R. Banse and K. R. Scherer, "Acoustic profiles in vocal emotion expression," J. Personality Soc. Psychol., vol. 70, no. 3, p. 614, 1996.
3. A. Beck and A. Hiolle et al., "Interpretation of emotional body language displayed by robots," in Proc. 3rd Int. Workshop Affective Interaction Natural Environments, 2010, pp. 37-42, ACM.
4. D. Bernhardt and P. Robinson, "Detecting emotions from connected action sequences," in IVIC '09. Berlin, Germany: Springer-Verlag, 2009, pp. 1-11.
5. M. Bhaykar and J. Yadav et al., "Speaker dependent, speaker independent, and cross language emotion recognition from speech using gmm, and hmm," in NCC. Piscataway, NJ, USA: IEEE, 2013, pp. 1-5.
6. C. M. Bishop and N. M. Nasrabadi, Pattern Recognition and Machine Learning. Berlin, Germany: Springer-Verlag, 2006, vol. 1.
7. S. Boucenna and P. Gaussier et al., "Imitation as a communication tool for online facial expression learning, and recognition," in Proc. IROS, 2010, pp. 5323-5328, IEEE.
8. C. L. Breazeal, Designing Sociable Robots. Cambridge, MA, USA: MIT press, 2004.
9. C. J. Burges, "A tutorial on support vector machines for pattern recognition," Data Mining Knowledge Discovery, vol. 2, no. 2, pp. 121-167, 1998. (Pubitemid 128695475)
10. F. Burkhardt and A. Paeschke et al., "A database of german emotional speech," in Interspeech, 2005, pp. 1517-1520. (Pubitemid 43908362)
11. J. J. Campos and D. I. Anderson et al., "Travel broadens the mind," Infancy, vol. 1, no. 2, pp. 149-219, 2000.
12. A. Camurri and G. Volpe et al., "Communicating expressiveness and affect in multimodal interactive systems," IEEE Trans. Multimedia, vol. 12, no. 1, pp. 43-53, Jan. 2005. (Pubitemid 40351838)
13. G. Castellano and S. D. Villalba et al., "Recognising human emotions from body movement, and gesture dynamics," in Proc. Affective Comput. Intell. Interaction, 2007, pp. 71-82, Springer.
14. J. J. Choi and Y. Kim et al., "Have you ever lied?: The impacts of gaze avoidance on people's perception of a robot," in HRI. Piscataway, NJ, USA: IEEE Press, 2013, pp. 105-106.
15. T. J. Clarke and M. F. Bradshaw et al., "The perception of emotion from bodymovement in point-light displays of interpersonal dialogue," Perception, vol. 34, no. 10, pp. 1171-1180, 2005. (Pubitemid 41659207)
16. M. Clynes, Sentics: The touch of emotions. NewYork,NY,USA: Anchor Press Garden, 1977.
17. R. Cowie and E. Douglas-Cowie et al., "Emotion recognition in human-computer interaction," IEEE Signal Process. Mag., vol. 18, no. 1, pp. 32-80, 2001. (Pubitemid 32287669)
18. A. P. Dempster and N. M. Laird et al., "Maximum likelihood from incomplete data via the em algorithm," J. Roy. Statist. Soc Series B (Methodological), vol. 39, no. 1, pp. 1-38, 1977.
19. A. Fernald, "Intonation and communicative intent in mothers' speech to infants: Is the melody the message?," Child Develop., vol. 60, no. 6, pp. 1497-1510, 1989.
20. A. Fernald, "Approval and disapproval: Infant responsiveness to vocal affect in familiar and unfamiliar languages," Child Develop., vol. 64, no. 3, pp. 657-674, 1993.
21. A. Fernald and T. Taeschner et al., "A cross-language study of prosodic modifications in mothers and fathers speech to preverbal infants," J. Child Lang., vol. 16, no. 3, pp. 477-501, 1989.
22. R. Fernandez and R.W. Picard, "Classical and novel discriminant features for affect recognition from speech," in Proc. Interspeech, 2005, pp. 473-476. (Pubitemid 43908102)
23. R. Flom and D. A. Gentile et al., "Infants discrimination of happy and sad music," Infant Behav. Develop., vol. 31, no. 4, pp. 716-728, 2008.
24. P. E. Gallaher, "Individual differences in nonverbal behavior: Dimensions of style," J. Personality Soc. Psychol., vol. 63, no. 1, p. 133, 1992.
25. T. Grossmann, "The development of emotion perception in face and voice during infancy," Restorative Neurol. Neurosci., vol. 28, no. 2, pp. 219-236, 2010.
26. T. Grossmann and T. Striano et al., "Crossmodal integration of emotional information from face and voice in the infant brain," Develop. Sci., vol. 9, no. 3, pp. 309-315, 2006. (Pubitemid 43572368)
27. H. Gunes and B. Schuller et al., "Emotion representation, analysis, and synthesis in continuous space: A survey," in FG. Piscataway, NJ, USA: IEEE, 2011, pp. 827-834.
28. D. Janssen and W. I. Schöllhorn et al., "Recognition of emotions in gait patterns by means of artificial neural nets," J. Nonverbal Behav., vol. 32, no. 2, pp. 79-92, 2008.
29. P. N. Juslin and P. Laukka, "Communication of emotions in vocal expression and music performance: Different channels, same code?," Psychol. Bulletin, vol. 129, no. 5, p. 770, 2003.
30. T. Kanungo and D. M. Mount et al., "An efficient k-means clustering algorithm: Analysis and implementation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 881-892, Jul. 2002. (Pubitemid 34835464)
31. M. Karg and R. Jenke et al., "A two-fold pca-approach for inter-individual recognition of emotions in natural walking," in Proc. MLDM Posters, 2009, pp. 51-61.
32. M. Karg and K. Kuhnlenz et al., "Recognition of affect based on gait patterns," IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 40, no. 4, pp. 1050-1061, Apr. 2010.
33. Y. E. Kim and E. M. Schmidt et al., "Music emotion recognition: A state of the art review," in Proc. ISMIR, 2010, pp. 255-266.
34. P. K. Kuhl, "Early language acquisition: Cracking the speech code," Nature Rev. Neurosci., vol. 5, no. 11, pp. 831-843, 2004. (Pubitemid 39482152)
35. K. Kuji, "Applicability of "rem", revised emotions measurement, to marketing analysis," Adv. Consumer Studies, vol. 15, pp. 57-76, 2009.
36. M. Lewis, "Self-conscious emotions," Handbook of Emotions, vol. 2, pp. 623-636, 2000.
37. M. M. Lewis, Infant speech; a study of the beginnings of language. London, U.K.: Routledge & Kegan Paul, 1936.
38. A. Lim and T. Ogata et al., "Converting emotional voice to motion for robot telepresence," in Humanoids. Piscataway, NJ, USA: IEEE, 2011, pp. 472-479.
39. A. Lim and T. Ogata et al., "Towards expressive musical robots: A cross-modal framework for emotional gesture, voice and music," EURASIP J. Audio, Speech, Music Process., vol. 2012, no. 1, pp. 1-12, 2012.
40. A. Lim and H. G. Okuno, "Using speech data to recognize emotion in human gait," in HBU. Berlin, Germany: Springer-Verlag, 2012, pp. 52-64.
41. H.-o. Lim and A. Ishii et al., "Emotion-based biped walking," Robotica, vol. 22, no. 5, pp. 577-586, 2004.
42. G. Littlewort and J. Whitehill et al., "The computer expression recognition toolbox (cert)," in FG. Piscataway, NJ, USA: IEEE, 2011, pp. 298-305.
43. S. R. Livingstone and R. Muhlberger et al., "Changing musical emotion: A computational rule system for modifying score and performance," Computer Music J., vol. 34, no. 1, pp. 41-64, 2010.
44. Y.Ma andH.M. Paterson et al., "Amotion capture library for the study of identity, gender, and emotion perception from biological motion," Behav. Res. Methods, vol. 38, no. 1, pp. 134-141, 2006. (Pubitemid 44945608)
45. M. Mancini and G. Castellano, "Real-time analysis, and synthesis of emotional gesture expressivity," in Proc. Doctoral Consortium Int. Conf. Affective Comput. Intell. Interaction, 2007.
46. A. Mehrabian, "Framework for a comprehensive description and measurement of emotional states," Genetic, Soc., General Psychol. Monographs, vol. 121, no. 3, pp. 339-361, 1995.
47. L. Mion and G.De Poli, "Score-independent audio features for description of music expression," IEEE Trans. Audio, Speech, Lang. Process., vol. 16, no. 2, pp. 458-466, 2008.
48. J. Montepare and E. Koff et al., "The use of body movements and gestures as cues to emotions in younger and older adults," J. Nonverbal Behav., vol. 23, no. 2, pp. 133-152, 1999.
49. J. M. Montepare and S. B. Goldstein et al., "The identification of emotions from gait information," J. Nonverbal Behav., vol. 11, no. 1, pp. 33-42, 1987.
50. K. Nakadai and T. Takahashi et al., "Design and implementation of robot audition system'hark' open source software for listening to three simultaneous speakers," Adv. Robot., vol. 24, no. 5-6, pp. 739-761, 2010.
51. D. G. K. Nelson and K. Hirsh-Pasek et al., "How the prosodic cues in motherese might assist language learning," J. Child Lang., vol. 16, no. 01, pp. 55-68, 1989.
52. A. Ortony, The cognitive structure of emotions. Cambridge,MA, USA: Cambridge Univ. Press, 1990.
53. P.-y. Oudeyer, "The synthesis of cartoon emotional speech," presented at the Speech Prosody 2002 Int. Conf., 2002.
54. F. Pedregosa and G. Varoquaux et al., "Scikit-learn: Machine learning in python," J. Mach. Learn. Res., vol. 12, pp. 2825-2830, 2011.
55. C. Pelachaud, "Studies on gesture expressivity for a virtual agent," Speech Commun., vol. 51, no. 7, pp. 630-639, 2009.
56. L. Peterson and M. J. Peterson, "Short-term retention of individual verbal items," J. Exp. Psychol., vol. 58, no. 3, p. 193, 1959.
57. F. E. Pollick and H. M. Paterson et al., "Perceiving affect from arm movement," Cognition, vol. 82, no. 2, pp. B51-B61, 2001. (Pubitemid 33041320)
58. T. Polzehl and A. Schmitt et al., "Approaching multi-lingual emotion recognition from speech-on language dependency of acoustic/prosodic features for anger detection," in Proc. Speech Prosody, 2010.
59. C. L. Roether and L. Omlor et al., "Critical features for the perception of emotion from gait," J. Vision, vol. 9, no. 6, 2009.
60. E. T. Rolls, "Precis of the brain and emotion," Behav. Brain Sci., vol. 23, no. 2, pp. 177-191, 2000.
61. J. A. Russell and J.-A. Bachorowski et al., "Facial and vocal expressions of emotion," Annu. Rev. Psychol., vol. 54, no. 1, pp. 329-349, 2003. (Pubitemid 37417497)
62. C. Saint-Georges and M. Chetouani et al., "Motherese in interaction: At the cross-road of emotion and cognition? (a systematic review)," PloS One, vol. 8, no. 10, p. e78103, 2013.
63. E. Scheiner and K. Hammerschmidt et al., "Acoustic analyses of developmental changes and emotional expression in the preverbal vocalizations of infants," J. Voice, vol. 16, no. 4, pp. 509-529, 2002. (Pubitemid 36636988)
64. K. R. Scherer, "Vocal affect expression: A review and a model for future research," Psychol. Bulletin, vol. 99, no. 2, p. 143, 1986.
65. K. R. Scherer, "A cross-cultural investigation of emotion inferences from voice and speech: Implications for speech technology," in Proc. Interspeech, 2000, pp. 379-382.
66. G. Schwarz, "Estimating the dimension of a model," Annal. Statist., vol. 6, no. 2, pp. 461-464, 1978.
67. P. Shen and Z. Changjun et al., "Automatic speech emotion recognition using support vector machine," inEMEIT. Piscataway, NJ,USA: IEEE Press, 2011, vol. 2, pp. 621-625.
68. B. Sievers and L. Polansky et al., "Music and movement share a dynamic structure that supports universal expressions of emotion," Proc. Nat. Acad. Sci., vol. 110, no. 1, pp. 70-75, 2013.
69. C. T. Snowdon, "Expression of emotion in non-human animals," in Handbook of Affective Sciences. London, U.K.: Oxford Univ. Press, 2003, pp. 457-480.
70. N. H. Soken and A. D. Pick, "Intermodal perception of happy and angry expressive behaviors by seven-month-old infants," Child Develop., vol. 63, no. 4, pp. 787-795, 1992.
71. P. Somervuo and T. Kohonen, "Self-organizing maps and learning vector quantization for feature sequences," Neural Process. Lett., vol. 10, no. 2, pp. 151-159, 1999. (Pubitemid 30503364)
72. F. Spector and D. Maurer, "Synesthesia: A new approach to understanding the development of perception," Develop. Psychol., vol. 45, no. 1, p. 175, 2009.
73. H. Spencer, "The origin and function of music," Frasers Mag., vol. 56, pp. 396-408, 1857.
74. L. J. Trainor and C. M. Austin et al., "Is infant-directed speech prosody a result of the vocal expression of emotion?," Psychol. Sci., vol. 11, no. 3, pp. 188-195, 2000.
75. M. Unuma and K. Anjyo et al., "Fourier principles for emotion-based human figure animation," in SIGGRAPH, 1995, pp. 91-96, ACM.
76. R. Van Bezooijen and S. A. Otto et al., "Recognition of vocal expressions of emotion a three-nation study to identify universal characteristics," J. Cross-Cultural Psychol., vol. 14, no. 4, pp. 387-406, 1983.
77. A. S. Walker-Andrews, "Infants' perception of expressive behaviors: Differentiation of multimodal information," Psychol. Bulletin, vol. 121, no. 3, p. 437, 1997.
78. A. Watanabe and M. Ogino et al., "Mapping facial expression to internal states based on intuitive parenting," J. Robot. Mechatron., vol. 19, no. 3, p. 315, 2007.
79. M. Zecca and Y. Mizoguchi et al., "Whole body emotion expressions for kobian humanoid robot-preliminary experiments with different emotional patterns," in RO-MAN. Piscataway, NJ, USA: IEEE Press, 2009, pp. 381-386.