A system for real-time multimodal analysis of nonverbal affective social interaction in user-centric media

IEEE Transactions on Multimedia

Volumn 12, Issue 6, 2010, Pages 576-590

  Varni, Giovanna ^a   Volpe, Gualtiero ^a   Camurri, Antonio ^a  

^a UNIVERSITY OF GENOA   (Italy)

Author keywords

Affective social behavior of small groups of users; affective sound and music processing; multimodal analysis of nonverbal affective social interaction; real time applications of multimodal affective social interaction; user centric media

Indexed keywords

AFFECTIVE SOUND AND MUSIC PROCESSING; MULTIMODAL ANALYSIS; REAL-TIME APPLICATION; SMALL GROUPS; USER-CENTRIC;

COMMUNICATION;

EID: 77956753635     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2010.2052592     Document Type: Article

References (65)

1. I. Laso-Ballesteros and P. Daras, User Centric Future Media, EU Commission, Sep. 2008.
2. N. Magnenat-Thalmann, A. Peternier, X. Righetti, M. Lim, G. Papagiannakis, T. Fragopoulos, K. Lambropoulou, P. Barsocchi, and D. Thalmann, "A virtual 3D mobile guide in the INTERMEDIA project," Vis. Comput., vol.24, no.7-9, pp. 827-836, Jul. 2008, Springer.
3. M. Vinyes, J. Bonada, and A. Loscos, "Demixing commercial music productions via human-assisted time-frequency masking," in Proc. 120th AES Conv., Paris, France, 2006.
4. A. Camurri, C. Canepa, P. Coletta, N. Ferrari, B. Mazzarino, and G. Volpe, "The interactive piece the bow is bent and drawn," in Proc. 3rd ACM Int. Conf. Digital Interactive Media in Entertainment and Arts (DIMEA 2008), Athens, Greece, Sep. 2008, pp. 376-383.
5. D. Thalmann and S. R. Musse, Crowd Simulation. New York: Springer-Verlag, 2007.
6. A. Vinciarelli, M. Pantic, and H. Bourlard, "Social signal processing: Survey of an emerging domain," Image Vis. Comput., vol.27, no.12, pp. 1743-1759, 2009.
7. A. Pentland, "Social signal processing," IEEE Signal Process. Mag., vol.24, no.4, pp. 108-111, Jul. 2007.
8. F. Pianesi, M. Zancanaro, E. Not, C. Leonardi, V. Falcon, and B. Lepri, "Multimodal support to group dynamics," Pers. Ubiq. Comput., vol.12, no.3, pp. 181-195, 2008.
9. I. McCowan, D. Gatica-Perez, S. Bengio, D. Moore, and H. Bourlard, "Towards computer understanding of human interactions," in Ambient Intelligence, E. Aarts, R. Collier, E. van Loenen, and B. de Ruyter, Eds. Heidelberg, Germany: Springer-Verlag, 2004, pp. 235-251.
10. D. Gatica-Perez, "Automatic nonverbal analysis of social interaction in small groups: A review," Image Vis. Comput., vol.27, no.12, pp. 1775-1787, Nov. 2009.
11. Z. Kasap, M. Benmoussa, P. Chaudhuri, and N. Magnenat-Thalmann, "Making them remember-Emotional virtual characters with memory," IEEE Comput. Graph. Appl., vol.29, no.2, pp. 20-29, Mar. 2009.
12. K. Liu and R. W. Picard, "Embedded empathy in continuous, interactive health assessment," in Proc. CHI Workshop HCI Challenges in Health Assessment, 2005.
13. J. Klein, "Computer response to user frustration," M.S. thesis, MIT, Cambridge, MA, 1999.
14. A. de Rosis, F. Cavalluzzi, I. Mazzotta, and N. Novielli, "Can embodied conversational agents induce empathy in users?," in Proc. Virtual Social Character Symp. (AISB 05), Apr. 2005.
15. S. W. McQuiggan and J. C. Lester, "Modeling and evaluating empathy in embodied companion agents," Int. J. Human-Computer Stud., vol.65, no.4, pp. 348-360, 2007.
16. A. Camurri, B. Mazzarino, M. Ricchetti, R. Timmers, and G. Volpe, "Multimodal analysis of expressive gesture in music and dance performances," in Gesture-Based Communication in Human Computer Interaction. Heidelberg, Germany: Springer-Verlag, 2004, vol.2915, LNAI.
17. A. Camurri, I. Lagerlöf, and G. Volpe, "Recognizing emotion from dance movement: Comparison of spectator recognition and automated techniques," Int. J. Human-Computer Stud., vol.59, pp. 213-225, Jul. 2003, Elsevier Science.
18. A. Camurri, B. Mazzarino, and G. Volpe, "The EyesWeb expressive gesture library," in Gesture-Based Communication in Human Computer Interaction. Heidelberg, Germany: Springer-Verlag, 2004, vol.2915, LNAI.
19. A. Camurri, G. Volpe, G. De Poli, and M. Leman, "Communicating expressiveness and affect in multimodal interactive systems," IEEE Multimedia Mag., vol.12, no.1, pp. 43-53, Jan.-Mar. 2005.
20. A. Kapur, A. Kapur, N. Virji-Babul, G. Tzanetakis, and P. F. Driessen, "Gesture-based affective computing on motion capture data. Lecture notes in computer science," in Affective Computing and Intelligent Interaction. Heidelberg, Germany: Springer-Verlag, 2005, vol.3784, LNCS.
21. J. Wagner, E. Andre, and F. Jung, "Smart sensor integration: A framework for multimodal emotion recognition in real-time," in Proc. 2009 Int. Conf. Affective Computing & Intelligent Interaction (ACII 2009), 2009.
22. L. Diduch, A. Fillingher, I. Hamchi, M. Horau, and V. Stanford, "Synchronization of data streams in distributed realtime multimodal environments using commodity hardware," in Proc. IEEE Int. Conf. Multimedia & Expo (ICME), 2008.
23. E. Douglas-Cowie, N. Campbell, R. Cowie, and P. Roach, "Emotional speech: Towards a newgeneration of databases," Speech Commun., vol.40, no.1-2, pp. 33-60, 2003.
24. S. Hashimoto, "KANSEI as the third target of information processing and related topics in Japan," in Proc. Int.Workshop KANSEI: The Technology of Emotion, 1997, pp. 101-104.
25. J. A. Russel, "A circumplex model of affect," J. Personal. Soc. Psychol., vol.39, pp. 1161-1178, 1980.
26. F. Takens, "Detecting strange attractors in turbolence," in Dynamical Systems and Turbulence, D. Rand and L.-S. Young, Eds. Heidelberg, Germany: Springer-Verlag, 1981, vol.898, LNM, pp. 366-381.
27. L. Cao, A. Mees, and K. Judd, "Dynamics from multivariate time series," Physica D, vol.121, pp. 75-88, 1998.
28. A. Vinciarelli, M. Pantic, H. Bourland, and A. Pentland, "Social signals, their function, and automatic analysis: A survey," in Proc. 10th Int. Conf. Multimodal Interfaces (ICMI08), Oct. 2008, pp. 61-68, ACM.
29. S. Boccaletti, J. Kurths, G. Osipov, D. L. Valladares, and C. S. Zhou, "The synchronisation of chaotic systems," Phys. Rep., vol.366, no.1-2, pp. 1-101, 2002.
30. M. C. Romano, M. Thiel, J. Kurths, I. Z. Kiss, and J. L. Hudson, "Detection of synchronisation for non-phase coherent and non-stationarity data," Europhys. Lett., vol.71, no.3, pp. 466-472, 2005.
31. J. P. Eckmann, S. O. Kamphorst, and D. Ruelle, "Recurrence plots of dynamical system," Europhys. Lett., vol.5, pp. 973-977, 1980.
32. N. Marwan, M. C. Romano, M. Thiel, and J. Kurths, "Recurrence plots for the analysis of complex systems," Phys. Rep., vol.438, no.237-329, 2007.
33. J. Nawrath, M. C. Romano, M. Thiel, I. Z. Kiss, M. Wickramasinghe, J. Timmer, J. Kurths, and B. Schelter, "Distinguish direct from indirect interactions in oscillatory networks with multiple time scales," Phys. Rev. Lett., vol.104, 038701, 2010.
34. R. Rienks and D. Heylen, "Automatic dominance detection in meetings using support vector machines," in Proc. Workshop Machine Learning for Multimodal Interaction, 2005.
35. N. E. Dunbar and J. Burgoon, "Perceptions of power and interactional dominance in interpersonal relationships," J. Social Person. Relationships, vol.22, no.2, pp. 207-233, 2005.
36. R. Norton, Communicator Style: Theory, Applications, and Measures. Beverly Hills, CA: Sage, 1983.
37. M. Schmid Mast, "Dominance as expressed and inferred through speaking time: A meta-analysis," Human Commun. Res., vol.28, no.3, pp. 420-450, Jul. 2002.
38. J. E. Driskell, G. F. Goodwin, E. Salas, and P. G. O'Shea, "What makes a good team player? Personality and team effectiveness," Group Dynam.: Theory, Res., Pract., vol.10, pp. 249-271, 2006.
39. A. Pentland, "Social Dynamics," in Proc. 3rd IEEE Int. Conf. Development and Learning (ICDL'04), 2004.
40. P. Arnhold, J. Grassberger, K. Lehnertz, and C. E. Elger, "A robust method for detecting interdependencies: Application to intracranially recorded EEG," Physica D, vol.134, no.4, pp. 419-430, 1999.
41. M. G. Rosenblum and A. S. Pikovsky, "Detecting direction of coupling in interacting oscillators," Phys. Rev. E, vol.64, no.045202, pp. 1-4, 2001.
42. R. Quian Quiroga, T. Kreuz, and P. Grassberger, "Event synchronisation: A simple and fast method to measure synchronicity and time delay patterns," Phys. Rev. E, vol.66, no.041904, pp. 1-9, 2002.
43. J. Kim and E. André, "Emotion recognition based on physiological changes in listening music," IEEE Trans. Pattern Anal. Mach. Intell., vol.30, no.12, pp. 2067-2083, Dec. 2008.
44. C. L. Krumhansl, "An exploratory study of musical emotions and psychophysiology," Can. J. Experiment. Psychol., vol.51, pp. 336-352, 1997.
45. J. Jaimovich and R. B. Knapp, "Pattern recognition of emotional states during musical performance from physiological signal," in Proc. 2009 Int. Computer Music Conf., Montreal, QC, Canada, Aug. 2009, pp. 461-464.
46. P. Kivy, Sound Sentiment: An Essay on the Musical Emotions. Philadelphia, PA: Temple Univ. Press, 1989.
47. J. Bhattacharya and H. Petsche, "Phase synchrony analysis of EEG during music perception reveals changes in functional connectivity due to musical expertise," Signal Process., vol.85, no.11, pp. 2161-2177, 2005.
48. J. P. Lachaux, E. Rodriguez, J. Martinerie, and F. Varela, "Measuring phase synchrony in brain signals," Human Brain Map., vol.8, pp. 194-208, 1999.
49. A. Pikovsky, M. G. Rosenblum, and J. Kurths, "Synchronization in a population of globally coupled chaotic oscillators," Europhys. Lett., vol.34, no.3, pp. 165-170, 1996.
50. Z. Neda, E. Ravasz, T. Vicsek, Y. Brechet, and A. L. Barabasi, "The sound of many hands clapping," Nature, vol.403, pp. 849-850, 2000.
51. D. C. Richardson, R. Dale, and N. Z. Kirkham, "The art of conversation is coordination: Common ground and the coupling of eye movements during dialogue," Psych. Sci., vol.18, no.5, pp. 407-413, 2007.
52. V. Trianni and S. Nolfi, "Self-organizing sync in a robotic swarm: A dynamical system view," IEEE Trans. Evol. Comput., vol.13, no.4, pp. 722-741, Aug. 2009.
53. R. Timmers, M. Marolt, A. Camurri, and G. Volpe, "Listeners' emotional engagement with performances of a skriabin étude: An explorative case study," Psychol. Music, vol.34, no.4, pp. 481-510, 2006.
54. G. Castellano, A. Mortillaro, M. Camurri, K. Scherer, and G. Volpe, "Automated analysis of body movement in emotionally expressive piano performances," Music Percep., vol.26, no.2, pp. 103-119, 2008.
55. J. W. Davidson, "What type of information is conveyed in the body movements of solo musician performer?," J. Human Move. Stud., vol.6, pp. 279-301, 1994.
56. J. P. Zbilut, J.-M. Zaldivar-Comenges, and F. Strozzi, "Recurrence quantification based on Lyapunov exponents for monitoring divergence in experimental data," Phys. Lett. A, vol.297, no.3-4, pp. 173-181, 2002.
57. M. Thiel, M. C. Romano, J. Kurths, M. Rolfs, and R. Kliegl, "Twin surrogates to test for complex synchronisation," Europhys. Lett., vol.75, no.4, pp. 535-541, 2006.
58. P. Keller, "Joint action in music performance," in Enacting Intersubjectivity: A Cognitive and Social Perspective to the Study of Interactions. Amsterdam, The Netherlands: IOS, 2008.
59. R. Laban and F. C. Lawrence, Effort. London, U.K.: Macdonald & Evans, 1947.
60. M. De Meijer, "The contribution of general features of body movement to the attribution of emotions," J. Nonverb. Behav., vol.13, pp. 247-268, 1989.
61. R. T. Boone and J. G. Cunningham, "Children's decoding of emotion in expressive body movement: The development of cue attunement," Development. Psychol., vol.34, pp. 1007-1016, 1998.
62. H. G. Wallbott, "Bodily expression of emotion," Eur. J. Social Psych., vol.28, pp. 879-896, 1998.
63. S. Dixon, "Automatic extraction of tempo and beat from expressive performances," J. New Music Res., vol.30, no.2, pp. 39-58, 2001.
64. G. Varni, M. Mancini, G. Volpe, and A. Camurri, "Sync'n'Move: Social interaction based on music and gesture," in Proc. 1st Int. ICST Conf. User Centric Media (UCMedia09).
65. G. Varni, A. Camurri, P. Coletta, and G. Volpe, "Toward real-time automated measure of empathy and dominance," in Proc. 2009 IEEE Int. Conf. Social Computing (SocialCom-09), Vancouver, BC, Canada, Aug. 2009.