Social signal processing: State-of-the-art and future perspectives of an emerging domain

MM'08 - Proceedings of the 2008 ACM International Conference on Multimedia, with co-located Symposium and Workshops

Volumn , Issue , 2008, Pages 1061-1070

  Vinciarelli, Alessandro ^a,b   Pantic, Maja ^c,d   Bourlard, Hervé ^a,b   Pentland, Alex ^e  

^a IDIAP RESEARCH INSTITUTE   (Switzerland)

^b EPFL   (Switzerland)

^c IMPERIAL COLLEGE LONDON   (United Kingdom)

^d UNIVERSITY OF TWENTE   (Netherlands)

^e MIT MEDIA LAB   (United States)

Author keywords

Algorithms

Indexed keywords

AUTOMATED SYSTEMS; DESIGN AND DEVELOPMENT; FUTURE PERSPECTIVES; HUMAN INTELLIGENCE; MACHINE ANALYSIS; SOCIAL BEHAVIOUR; SOCIAL INTELLIGENCE; SOCIAL PSYCHOLOGY; SOCIAL SIGNALS; SOCIALLY-AWARE COMPUTING;

COMPUTER SCIENCE; MACHINE DESIGN; METHOD OF MOMENTS; MULTIMEDIA SYSTEMS;

SIGNAL PROCESSING;

EID: 70350619805     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1459359.1459573     Document Type: Conference Paper

References (98)

1. P. Aarabi, D. Hughes, K. Mohajer, and M. Emami. The automatic measurement of facial beauty. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics, pages 2644-2647, 2001.
2. K. Albrecht. Social Intelligence: The new science of success. John Wiley & Sons Ltd, 2005.
3. N. Ambady and R. Rosenthal. Thin slices of expressive behavior as predictors of interpersonal consequences: a meta-analysis. Psychological Bulletin, 111(2):256-274, 1992.
4. J. Bailenson, N. Yee, K. Patel, and A. Beall. Detecting digital chameleons. Computers in Human Behavior, 24(1):66-87, 2008.
5. R. Bales. Interaction Process Analysis: A Method for the Study of Small Groups. Addison-Wesley, 1950.
6. S. Banerjee and A. Rudnicky. Using simple speech based features to detect the state of a meeting and the roles of the meeting participants. In Proceedings of International Conference on Spoken Language Processing, pages 2189-2192, 2004.
7. R. Barzilay, M. Collins, J. Hirschberg, and S. Whittaker. The rules behind the roles: identifying speaker roles in radio broadcasts. In Proceedings of American Association of Artificial Intelligence Symposium, pages 679-684, 2000.
8. P. Boersma and D. Weenink. Praat, a system for doing phonetics by computer. Glot International, 5(9/10):341-345, 2001.
9. L. Bretzner, I. Laptev, and T. Lindeberg. Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, pages 405-410, 2002.
10. M. Buchanan. The science of subtle signals. Strategy+Business, 48:68-77, 2007.
11. N. Campbell. Conversational speech synthesis and the need for some laughter. IEEE Transactions on Speech and Language Processing, 14(4):1171-1178, 2006.
12. T. Chartrand and J. Bargh. The chameleon effect: the perception-behavior link and social interaction. Journal of Personality and Social Psychology, 76(6):893-910, 1999.
13. J. Cohn. Foundations of human computing: facial expression and emotion. In Proceedings of the ACM International Conference on Multimodal Interfaces, pages 233-238, 2006.
14. J. Cohn and P. Ekman. Measuring facial action by manual coding, facial EMG, and automatic facial image analysis. In J. Harrigan, R. Rosenthal, and K. Scherer, editors, Handbook of nonverbal behavior research methods in the affective sciences, pages 9-64. 2005.
15. J. Cortes and F. Gatti. Physique and self-description of temperament. Journal of Consulting Psychology, 29(5):432-439, 1965.
16. D. Crystal. Prosodic Systems and Intonation in English. Cambridge University Press, 1969.
17. D. Cunningham, M. Kleiner, H. Bülthoff, and C. Wallraven. The components of conversational facial expressions. Proceedings of the Symposium on Applied Perception in Graphics and Visualization, pages 143-150, 2004.
18. J. Curhan and A. Pentland. Thin slices of negotiation: predicting outcomes from conversational dynamics within the first 5 minutes. Journal of Applied Psychology, 92(3):802-811, 2007.
19. T. Darrell, G. Gordon, M. Harville, and J. Woodfill. Integrated person tracking using stereo, color, and pattern detection. International Journal of Computer Vision, 37(2):175-185, 2000.
20. R. De Silva and N. Bianchi-Berthouze. Modeling human affective postures: an information theoretic characterization of posture features. Journal of Computational Animation and Virtual World, 15(3-4):269-276, 2004.
21. A. Dielmann and S. Renals. Automatic meeting segmentation using dynamic bayesian networks. IEEE Transactions on Multimedia, 9(1):25, 2007.
22. K. Dion, E. Berscheid, and E. Walster. What is beautiful is good. Journal of Personality and Social Psychology, 24(3):285-290, 1972.
23. W. Dong, B. Lepri, A. Cappelletti, A. Pentland, F. Pianesi, and M. Zancanaro. Using the influence model to recognize functional roles in meetings. In Proceedings of the International Conference on Multimodal Interfaces, pages 271-278, 2007.
24. N. Eagle and A. Pentland. Reality mining: sensing complex social signals. Journal of Personal and Ubiquitous Computing, 10(4):255-268, 2006.
25. Y. Eisenthal, G. Dror, and E. Ruppin. Facial attractiveness: Beauty and the machine. Neural Computation, 18(1):119-142, 2005.
26. P. Ekman and W. Friesen. Facial action coding system (FACS): Manual. 2002.
27. P. Ekman and E. Rosenberg. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, 2005.
28. N. Fragopanagos and J. Taylor. Emotion recognition in human-computer interaction. Neural Networks, 18(4):389-405, 2005.
29. H. Giles, N. Coupland, and J. Coupland. Accomodation theory: communication, context, consequence. In Contexts of accomodation: developments in applied sociolinguistics, pages 1-69. Cambridge University Press, 1991.
30. M. Gladwell. Blink: The Power of Thinking without Thinking. Little Brown & Company, 2005.
31. C. Glass, T. Merluzzi, J. Biever, and K. Larsen. Cognitive assessment of social anxiety: Development and validation of a self-statement questionnaire. Cognitive Therapy and Research, 6(1):37-55, 1982.
32. K. Greene. 10 emerging technologies 2008. MIT Technology Review, February 2008.
33. S. Gregory, K. Dagan, and S. Webster. Evaluating the relation of vocal accommodation in conversation partners fundamental frequencies to perceptions of communication quality. Journal of Nonverbal Behavior, 21(1):23-43, 1997.
34. E. Hall. The silent language. Doubleday, 1959.
35. X. Huang, A. Acero, and H. Hon. Spoken language processing. Prentice Hall, 2001.
36. H. Hung, D. Jayagopi, C. Yeo, G. Friedland, S. Ba, J. Odobez, K. Ramchandran, N. Mirghafori, and D. Gatica-Perez. Using audio and video features to classify the most dominant person in a group meeting. In Proceedings of the ACM International Conference on Multimedia, pages 835-838, 2007.
37. A. Ito, X. Wang, M. Suzuki, and S. Makino. Smile and laughter recognition using speech processing and face recognition from conversation video. In Proceedings of th International Conference on Cyberworlds, pages 437-444, 2005.
38. A. Jaimes, K. Omura, T. Nagamine, and K. Hirata. Memory cues for meeting video retrieval. In Proceedings of Workshop on Continuous Archival and Retrieval of Personal Experiences, pages 74-85, 2004.
39. D. Keltner and P. Ekman. Facial expression of emotion. In M. Lewis and J. Haviland-Jones, editors, Handbook of Emotions, pages 236-249. 2000.
40. D. Keltner and J. Haidt. Social functions of emotions at four levels of analysis. Cognition and Emotion, 13(5):505-521, 1999.
41. L. Kennedy and D. Ellis. Laughter detection in meetings. In Proceedings of the NIST Meeting Recognition Workshop, 2004.
42. M. Kimura and I. Daibo. Interactional synchrony in conversations about emotional episodes: A measurement by "the between-participants pseudosynchrony experimental paradigm". Journal of Nonverbal Behavior, 30(3):115-126, 2006.
43. A. Kleinsmith, R. De Silva, and N. Bianchi-Berthouze. Cross-cultural differences in recognizing affect from body posture. Interacting with Computers, 18(6):1371-1389, 2006.
44. M. Knapp and J. Hall. Nonverbal Communication in Human Interaction. Harcourt Brace College Publishers, 1972.
45. J. Lakin, V. Jefferis, C. Cheng, and T. Chartrand. The Chameleon effect as social glue: Evidence for the evolutionary significance of nonconscious mimicry. Journal of Nonverbal Behavior, 27(3):145-162, 2003.
46. Y. Li, S. Ma, and H. Lu. Human posture recognition using multi-scale morphological method and Kalman motion estimation. In Proceedings of International Conference on Pattern Recognition, pages 175-177, 1998.
47. G. Littlewort, M. Bartlett, and K. Lee. Faces of pain: automated measurement of spontaneous facial expressions of genuine and posed pain. In Proceedings of the International Conference on Multimodal Interfaces, pages 15-21, 2007.
48. S. Lucey, A. Ashraf, and J. Cohn. Investigating spontaneous facial action recognition through AAM representations of the face. In K. Delac and M. Grgic, editors, Handbook of Face Recognition, pages 275-286. I-Tech Education and Publishing, 2007.
49. I. McCowan, D. Gatica-Perez, S. Bengio, G. Lathoud, M. Barnard, and D. Zhang. Automatic analysis of multimodal group actions in meetings. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(3):305-317, 2005.
50. A. Mehrabian and S. Ferris. Inference of attitude from nonverbal communication in two channels. Journal of Counseling Psychology, 31(3):248-252, 1967.
51. T. Moeslund and E. Granum. A survey of computer vision-based human motion capture. Computer Vision and Image Understanding, 81(3):231-268, 2001.
52. N. Morgan and E. Fosler-Lussier. Combining multiple estimators of speaking rate. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, 1998.
53. S. Mota and R. Picard. Automated posture analysis for detecting learner's interest level. In Proceedings of Conference on Computer Vision and Pattern Recognition, pages 49-56, 2003.
54. K. Murphy. Dynamic Bayesian Networks: Representation, Inference and Learning. PhD thesis, University of California Berkeley, 2002.
55. C. Nass and K. Lee. Does computer-synthesized speech manifest personality? Experimental tests of recognition, similarity-attraction, and consistency-attraction. Journal of Experimental Psychology: Applied, 7(3):171-181, 2001.
56. A. Oikonomopoulos, M. Pantic, and I. Patras. B-spline polynomial descriptors for human activity recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2008.
57. N. Oliver, E. Horvitz, and A. Garg. Layered representations for human activity recognition. In Proceedings of the International Conference on Multimodal Interfaces, pages 3-8, 2002.
58. A. J. O'Toole, T. Price, T. Vetter, J. Bartlett, and V. Blanz. 3D shape and 2D surface textures of human faces: the role of "average" in attractiveness and age. Image and Vision Computing, 18(1):9-19, 1999.
59. S. Oviatt, C. Darves, and R. Coulston. Toward adaptive conversational interfaces: Modeling speech convergence with animated personas. ACM Transactions on Computer-Human Interaction, 11(3):300-328, 2004.
60. L. Ozer and W. Wolf. Real-time posture and activity recognition. In Proceedings of Workshop on Motion and Video Computing, pages 133-138, 2002.
61. M. Pantic and M. Bartlett. Machine analysis of facial expressions. In K. Delac and M. Grgic, editors, Handbook of Face Recognition, pages 377-416. I-Tech Education and Publishing, 2007.
62. M. Pantic, A. Pentland, A. Nijholt, and T. Huang. Human-centred intelligent human-computer interaction (HCI2): How far are we from attaining it? International Journal of Autonomous and Adaptive Communications Systems, 1(2):168-187, 2008.
63. M. Pantic and L. Rothkrantz. Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91(9):1370-1390, 2003.
64. C. Pelachaud, V. Carofiglio, B. De Carolis, F. de Rosis, and I. Poggi. Embodied contextual agent in information delivering application. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, pages 758-765, 2002.
65. A. Pentland. Social dynamics: Signals and behavior. In International Conference on Developmental Learning, 2004.
66. A. Pentland. Socially aware computation and communication. IEEE Computer, 38(3):33-40, 2005.
67. A. Pentland. Automatic mapping and modeling of human networks. Physica A, 378:59-67, 2007.
68. A. Pentland. Social signal processing. IEEE Signal Processing Magazine, 24(4):108-111, 2007.
69. S. Petridis and M. Pantic. Audiovisual discrimination between laughter and speech. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 5117-5121, 2008.
70. T. Pfau and G. Ruske. Estimating the speaking rate by vowel detection. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 945-948, 1998.
71. R. Pope. Vision-based human motion analysis: An overview. Computer Vision and Image Understanding, 108(1-2):4-18, 2007.
72. G. Psathas. Conversation Analysis - The study of talk-in-interaction. Sage Publications, 1995.
73. V. Richmond and J. McCroskey. Nonverbal Behaviors in interpersonal relations. Allyn and Bacon, 1995.
74. R. Rienks, D. Zhang, and D. Gatica-Perez. Detection and application of influence rankings in small group meetings. In Proceedings of the International Conference on Multimodal Interfaces, pages 257-264, 2006.
75. J. Russell, J. Bachorowski, and J. Fernandez-Dols. Facial and vocal expressions of emotion. Annual Reviews in Psychology, 54(1):329-349, 2003.
76. A. Scheflen. The significance of posture in communication systems. Psychiatry, 27:316-331, 1964.
77. K. Scherer. Personality markers in speech. Cambridge University Press, 1979.
78. K. Scherer. Vocal communication of emotion: A review of research paradigms. Speech Communication, 40(1-2):227-256, 2003.
79. B. Schuller, R. Müeller, B. Höernler, A. Höethker, H. Konosu, and G. Rigoll. Audiovisual recognition of spontaneous interest within conversations. In Proceedings of the International Conference on Multimodal Interfaces, pages 30-37, 2007.
80. E. Shriberg. Phonetic consequences of speech disfluency. Proceedings of the International Congress of Phonetic Sciences, 1:619-622, 1999.
81. E. Shriberg, A. Stolcke, and D. Baron. Observations of overlap: findings and implications for automatic processing of multiparty conversation. In Proceedings of Eurospeech, pages 1359-1362, 2001.
82. P. Shrout and D. Fiske. Nonverbal behaviors and social evaluation. Journal of Personality, 49(2):115-128, 1981.
83. K. Sjölander and J. Beskow. Wavesurfer-an open source speech tool. In Proceedings of International Conference on Spoken Language Processing, pages 464-467, 2000.
84. L. Smith-Lovin and C. Brody. Interruptions in group discussions: the effects of gender and group composition. American Sociological Review, 54(3):424-435, 1989.
85. H. Tischler. Introduction to Sociology. Harcourt Brace College Publishers, 1990.
86. Y. Tong, W. Liao, and Q. Ji. Facial action unit recognition by exploiting their dynamic and semantic relationships. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(10):1683-1699, 2007.
87. S. Tranter and D. Reynolds. An overview of automatic speaker diarization systems. IEEE Transactions on Audio, Speech, and Language Processing, 14(5):1557-1565, 2006.
88. K. Truong and D. van Leeuwen. Automatic discrimination between laughter and speech. Speech Communication, 49(2):144-158, 2007.
89. M. Valstar, H. Gunes, and M. Pantic. How to distinguish posed from spontaneous smiles using geometric features. In Proceedings of the International Conference on Multimodal Interfaces, pages 38-45, 2007.
90. A. Vinciarelli. Speakers role recognition in multiparty audio recordings using social network analysis and duration distribution modeling. IEEE Transactions on Multimedia, 9(9):1215-1226, 2007.
91. S. Wasserman and K. Faust. Social Network Analysis: Methods and Applications. Cambridge University Press, 1994.
92. C. Weng, W. Chu, and J. Wu. Movie analysis based on roles social network. In proceedings of IEEE International Conference on Multimedia and Expo, pages 1403-1406, 2007.
93. A. Williams. Facial expression of pain: An evolutionary account. Behavioral and Brain Sciences, 25(4):439-455, 2003.
94. M. Yang, D. Kriegman, and N. Ahuja. Detecting faces in images: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(1):34-58, 2002.
95. G. Yule. Pragmatics. Oxford University Press, 1996.
96. M. Zancanaro, B. Lepri, and F. Pianesi. Automatic detection of group functional roles in face to face interactions. In Proceedings of the International Conference on Multimodal Interfaces, pages 28-34, 2006.
97. B. Zellner. Pauses and the temporal structure of speech. In E. Keller, editor, Fundamentals of speech synthesis and speech recognition, pages 41-62. John Wiley & Sons, 1994.
98. Z. Zeng, M. Pantic, G. Roisman, and T. Huang. A survey of affect recognition methods: audio, visual and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008.