Towards expressive musical robots: A cross-modal framework for emotional gesture, voice and music

Eurasip Journal on Audio, Speech, and Music Processing

Volumn 2012, Issue 1, 2012, Pages

  Lim, Angelica ^a   Ogata, Tetsuya ^a   Okuno, Hiroshi G ^a  

^a KYOTO UNIVERSITY   (Japan)

Author keywords

Affective computing; Entertainment robots; Gesture

Indexed keywords

AFFECTIVE COMPUTING; CROSS-MODAL; DANCE STEPS; EMOTIONAL STATE; ENTERTAINMENT ROBOTS; GESTURE; MULTI-MODAL; MUSICAL PHRASE;

ROBOTS;

EID: 84873841216     PISSN: 16874714     EISSN: 16874722     Source Type: Journal    
DOI: 10.1186/1687-4722-2012-3     Document Type: Article

References (39)

1. G Weinberg, A Raman, T Mallikarjuna, Interactive jamming with Shimon: a social robotic musician, in Proceedings of HRI, San Diego, 233-234 (2009)
2. E Singer, J Feddersen, C Redmon, B Bowen, LEMUR's Musical Robots, in Proceedings of NIME, Hamamatsu, 181-184 (2004)
3. J Solis, K Chida, K Taniguchi, SM Hashimoto, K Suefuji, A Takanishi, The Waseda flutist robot WF-4RII in comparison with a professional flutist. Comput Music J. 30(4), 12-27 (2006). doi:10.1162/comj.2006.30.4.12
4. M Clynes, Sentics: The Touch of the Emotions (Prism Press, UK, 1989)
5. P Laukka, Instrumental music teachers' views on expressivity: a report from music conservatoires. Music Educ Res. 6(1), 45-56 (2004). doi:10.1080/1461380032000182821
6. PN Juslin, J Sloboda, Handbook of Music and Emotion, 1st edn. (Oxford University Press, USA, 2010)
7. E Lindstrom, PN Juslin, R Bresin, A Williamon, Expressivity comes from within your soul: a questionnaire study of music students' perspectives on expressivity. Res Stud Music Educ. 20(1), 23-47 (2003)
8. N Silvestrini, V Piguet, C Cedraschi, MR Zentner, Music and auditory distraction reduce pain: emotional or attentional effects? Music Med. 3, 264-270 (2011). doi:10.1177/1943862111414433
9. T Ogata, A Shimura, K Shibuya, S Sugano, A violin playing algorithm considering the change of phrase impression, in Proceedings of SMC, vol. 2. Nashville, 1342-1347 (2000)
10. J Solis, K Suefuji, K Taniguchi, T Ninomiya, M Maeda, A Takanishi, Implementation of expressive performance rules on the WF-4RIII by modeling a professional flutist performance using NN, in Proceedings of ICRA, Rome, 2552-2557 (2007)
11. T Nakano, M Goto, VocaListener: a singing-to-singing synthesis system based on iterative parameter estimation, in Proceedings of SMC, Porto, 343-348 (2009)
12. A Lim, T Mizumoto, T Takahashi, T Ogata, HG Okuno, Programming by playing and approaches for expressive robot performances, in IROS Workshop on Robots and Musical Expressions, Taipei (2010)
13. Y Kusuda, Toyota's violin-playing robot. Ind Robot: Int J. 35(6), 504-506 (2008). doi:10.1108/01439910810909493
14. A Kirke, ER Miranda, A survey of computer systems for expressive music performance. ACM Comput Surv. 42(1), 1-41 (2009)
15. WF Thompson, P Graham, FA Russo, Seeing music performance: visual influences on perception and experience. Semiotica. 156(1), 203-227 (2005)
16. M Schutz, M Kubovy, Seeing music: The percussionists' strategic use of gesture in live performance, in Proceedings of Music and Gesture, Manchester, 148-150 (2006)
17. M Zecca, Y Mizoguchi, K Endo, F Iida, Y Kawabata, Whole body Emotion Expressions for KOBIAN Humanoid Robot-preliminary experiments with different emotional patterns, in Proceedings of ROMAN, Toyama, 381-386 (2009)
18. PN Juslin, P Laukka, Communication of emotions in vocal expression and music performance: different channels, same code? Psychol Bull. 129(5), 770-814 (2003)
19. CL Breazeal, Designing sociable robots, 1st edn. (The MIT Press, Cambridge, MA, 2004)
20. R Cowie, E Douglas-Cowie, N Tsapatsoulis, G Votsis, S Kollias, W Fellenz, JG Taylor, Emotion Recognition in Human Computer Interaction. IEEE Signal Process Mag. 22(1), 33-80 (2005)
21. R Fernandez, RW Picard, Classical and novel discriminant features for affect recognition from speech, in Proceedings of Interspeech, Lisbon, 4-8 (2005)
22. Z Zeng, M Pantic, A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell. 31(1), 39-58 (2009)
23. JM Fellous, From human emotions to robot emotions, in AAAI Spring Symposium on Architectures for Modeling Emotion: Cross-Disciplinary Foundations, Palo Alto, 39-46 (2004)
24. L Mion, G De Poli, Score-independent audio features for description of music expression. IEEE Trans Audio Speech Lang Process. 16(2), 458-466 (2008)
25. H Gunes, B Schuller, M Pantic, R Cowie, Emotion representation, analysis and synthesis in continuous space: a survey, in Proceedings of FG, Santa Barbara, 827-834 (2011)
26. SR Livingstone, AR Brown, R Muhlberger, WF Thompson, Modifying score and performance changing musical emotion: a computational rule system for modifying score and performance. Comput Music J. 34(1), 41-65 (2010). doi:10.1162/comj. 2010.34.1.41
27. C Pelachaud, Studies on gesture expressivity for a virtual agent. Speech Commun. 51(7), 630-639 (2009). doi:10.1016/j.specom.2008.04.009
28. A Camurri, G Volpe, Communicating expressiveness and affect in multimodal interactive systems. Multimedia. 12(1), 43-53 (2005). doi:10.1109/MMUL.2005.2
29. A Lim, T Mizumoto, LK Cahier, T Otsuka, T Takahashi, K Komatani, T Ogata, HG Okuno, Robot musical accompaniment: integrating audio and visual cues for real-time synchronization with a human flutist, in Proceedings of IROS, Taipei, 1964-1969 (2010)
30. A Lim, T Ogata, HG Okuno, Converting emotional voice to motion for robot telepresence, in Humanoids, Bled, 472-479 (2011)
31. T Mizumoto, H Tsujino, T Takahashi, T Ogata, HG Okuno, Thereminist robot: development of a robot theremin player with feedforward and feedback arm control based on a Theremin's pitch model, in Proceedings of ICRA, Kobe, 2297-2302 (2009)
32. H Kenmochi, H Ohshita, Vocaloid-commercial singing synthesizer based on sample concatenation, in Proceedings of Interspeech, Antwerp, 4011-4010 (2007)
33. PN Juslin, Can results from studies of perceived expression in musical performances be generalized across response formats?. Psychomusicology 16, 77-101 (1999)
34. L Kessous, G Castellano, G Caridakis, Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis. J Multimodal User Interfaces 3(1-2), 33-48 (2009)
35. M Mancini, G Castellano, Real-time analysis and synthesis of emotional gesture expressivity, in Proc of the Doctoral Consortium of Intl Conf on Affective Computing and Intelligent Interaction, Lisbon (2007)
36. PE Gallaher, Individual differences in nonverbal behavior: dimensions of style. J Personal Soc Psychol. 63(1), 133-145 (1992)
37. HG Wallbott, Bodily expression of emotion. Europ J Soc Psychol. 28(6), 879-896 (1998). doi:10.1002/(SICI)1099-0992(1998110)28:63.0.CO;2-W
38. K Amaya, A Bruderlin, T Calvert, Emotion from motion. Graph Interface 1999 222-229 (1996).
39. FE Pollick, HM Paterson, A Bruderlin, AJ Sanford, Perceiving affect from arm movement. J Personal. 82, 51-61 (2001)